Project - PART 1

In [7]:
import tensorflow as tf
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import os
In [8]:
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import MinMaxScaler
from sklearn.linear_model import LinearRegression,Lasso,Ridge
from sklearn.ensemble import RandomForestRegressor,GradientBoostingRegressor
from sklearn.svm import SVR
from sklearn.neighbors import KNeighborsRegressor
from sklearn.metrics import mean_squared_error,r2_score,median_absolute_error
In [9]:
# Neural Network
from tensorflow.keras.models import Sequential
from tensorflow.keras import layers
from tensorflow.keras.layers import Dense, Activation
from tensorflow.keras import losses
from tensorflow.keras.losses import MAE,MSE
from tensorflow.keras import optimizers
from tensorflow.keras.optimizers import SGD,Adam

•DOMAIN: Electronics and Telecommunication

•CONTEXT: A communications equipment manufacturing company has a product which is responsible for emitting informative signals. Company wants to build a machine learning model which can help the company to predict the equipment’s signal quality using various parameters.

•DATA DESCRIPTION:</br> The data set contains information on various signal tests performed:</br> 1.Parameters: Various measurable signal parameters.</br> 2.Signal_Quality: Final signal strength or quality

•PROJECT OBJECTIVE: The need is to build a regressor which can use these parameters to determine the signal strength or quality [as number]

Project-PART 2

In [10]:
# importing dataset
data = pd.read_csv('Signal.csv')
data.head()
Out[10]:
Parameter 1 Parameter 2 Parameter 3 Parameter 4 Parameter 5 Parameter 6 Parameter 7 Parameter 8 Parameter 9 Parameter 10 Parameter 11 Signal_Strength
0 7.4 0.70 0.00 1.9 0.076 11.0 34.0 0.9978 3.51 0.56 9.4 5
1 7.8 0.88 0.00 2.6 0.098 25.0 67.0 0.9968 3.20 0.68 9.8 5
2 7.8 0.76 0.04 2.3 0.092 15.0 54.0 0.9970 3.26 0.65 9.8 5
3 11.2 0.28 0.56 1.9 0.075 17.0 60.0 0.9980 3.16 0.58 9.8 6
4 7.4 0.70 0.00 1.9 0.076 11.0 34.0 0.9978 3.51 0.56 9.4 5
In [11]:
data.info()
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 1599 entries, 0 to 1598
Data columns (total 12 columns):
 #   Column           Non-Null Count  Dtype  
---  ------           --------------  -----  
 0   Parameter 1      1599 non-null   float64
 1   Parameter 2      1599 non-null   float64
 2   Parameter 3      1599 non-null   float64
 3   Parameter 4      1599 non-null   float64
 4   Parameter 5      1599 non-null   float64
 5   Parameter 6      1599 non-null   float64
 6   Parameter 7      1599 non-null   float64
 7   Parameter 8      1599 non-null   float64
 8   Parameter 9      1599 non-null   float64
 9   Parameter 10     1599 non-null   float64
 10  Parameter 11     1599 non-null   float64
 11  Signal_Strength  1599 non-null   int64  
dtypes: float64(11), int64(1)
memory usage: 150.0 KB
In [12]:
data.describe().T
Out[12]:
count mean std min 25% 50% 75% max
Parameter 1 1599.0 8.319637 1.741096 4.60000 7.1000 7.90000 9.200000 15.90000
Parameter 2 1599.0 0.527821 0.179060 0.12000 0.3900 0.52000 0.640000 1.58000
Parameter 3 1599.0 0.270976 0.194801 0.00000 0.0900 0.26000 0.420000 1.00000
Parameter 4 1599.0 2.538806 1.409928 0.90000 1.9000 2.20000 2.600000 15.50000
Parameter 5 1599.0 0.087467 0.047065 0.01200 0.0700 0.07900 0.090000 0.61100
Parameter 6 1599.0 15.874922 10.460157 1.00000 7.0000 14.00000 21.000000 72.00000
Parameter 7 1599.0 46.467792 32.895324 6.00000 22.0000 38.00000 62.000000 289.00000
Parameter 8 1599.0 0.996747 0.001887 0.99007 0.9956 0.99675 0.997835 1.00369
Parameter 9 1599.0 3.311113 0.154386 2.74000 3.2100 3.31000 3.400000 4.01000
Parameter 10 1599.0 0.658149 0.169507 0.33000 0.5500 0.62000 0.730000 2.00000
Parameter 11 1599.0 10.422983 1.065668 8.40000 9.5000 10.20000 11.100000 14.90000
Signal_Strength 1599.0 5.636023 0.807569 3.00000 5.0000 6.00000 6.000000 8.00000
In [13]:
data.isnull().sum()
Out[13]:
Parameter 1        0
Parameter 2        0
Parameter 3        0
Parameter 4        0
Parameter 5        0
Parameter 6        0
Parameter 7        0
Parameter 8        0
Parameter 9        0
Parameter 10       0
Parameter 11       0
Signal_Strength    0
dtype: int64
In [14]:
data['Signal_Strength'].unique()
Out[14]:
array([5, 6, 7, 4, 8, 3])
In [15]:
plt.figure(figsize = (15,15))
for i in range(1,12):
    plt.subplot(3,4,i)
    sns.boxplot(x = data.columns[i],data = data)
plt.tight_layout()
plt.show()
In [16]:
plt.figure(figsize = (15,10))
for i in range(0,12):
    #plt.figure(figsize = (15,15))
    plt.subplot(3,4,i+1)
    col = data.columns[i]
    sns.kdeplot(x = col,data = data)
plt.tight_layout()
plt.show()

outcomes

  • Parameter 4,5,6,7,10 are right skewed
  • Parameter 8,9 are symmetric
  • Parameter 2 has two picks
  • Parameter 3 has 3 picks
In [17]:
for i in range(0,12):
    plt.figure(figsize = (15,15))
    plt.subplot(3,4,i+1)
    col = data.columns[i]
    sns.boxplot(y = col,x = 'Signal_Strength',data = data)
    plt.show()

Outcomes :

  • Higher the Parameter 2 lower the signal strength so we can see some negitivce corelation here
  • Higher the Parameter 3 higher the signal strength so we can see some positive corelation here
  • Higher the Parameter 10 higher the signal strength so we can see some positive corelation here
In [18]:
#plt.figure(figsize = (15,10))
for i in range(0,12):
    plt.figure(figsize = (15,15))
    plt.subplot(3,4,i+1)
    col = data.columns[i]
    if col != 'Signal_Strength':
        sns.kdeplot(x = col,hue = 'Signal_Strength',data = data)
    plt.tight_layout()
    plt.show()
  • Parameter 4, 5 are more skwed twards right that means has more outlier or values on right side
In [19]:
plt.figure(figsize = (10,10))
sns.heatmap(data.corr(),annot  = True,square = True,cbar = False)
plt.show()
In [20]:
plt.figure(figsize = (10,10))
sns.pairplot(data,hue = 'Signal_Strength')
plt.show()
<Figure size 720x720 with 0 Axes>
In [21]:
# Data Preprocessing
scaler = MinMaxScaler()
scaled_data  = pd.DataFrame(scaler.fit_transform(data),columns = data.columns)
X = scaled_data.iloc[:,:-1]
y = scaled_data.iloc[:,-1]

X_train,X_test,y_train,y_test  = train_test_split(X,y,test_size=0.2,stratify=y,random_state=1)
In [22]:
X_train
Out[22]:
Parameter 1 Parameter 2 Parameter 3 Parameter 4 Parameter 5 Parameter 6 Parameter 7 Parameter 8 Parameter 9 Parameter 10 Parameter 11
952 0.318584 0.130137 0.40 0.089041 0.076795 0.070423 0.014134 0.388399 0.448819 0.209581 0.430769
563 0.265487 0.116438 0.49 0.123288 0.133556 0.338028 0.190813 0.516153 0.448819 0.167665 0.261538
150 0.238938 0.143836 0.47 0.082192 0.108514 0.056338 0.017668 0.420705 0.464567 0.119760 0.292308
202 0.194690 0.260274 0.11 0.041096 0.105175 0.211268 0.151943 0.395007 0.488189 0.275449 0.169231
638 0.274336 0.544521 0.12 0.089041 0.218698 0.084507 0.060071 0.464758 0.480315 0.191617 0.276923
... ... ... ... ... ... ... ... ... ... ... ...
1119 0.088496 0.369863 0.00 0.109589 0.090150 0.084507 0.031802 0.182819 0.614173 0.149701 0.692308
80 0.141593 0.226027 0.20 0.047945 0.095159 0.028169 0.031802 0.420705 0.527559 0.137725 0.123077
1474 0.469027 0.260274 0.50 0.883562 0.322204 0.661972 0.268551 0.906755 0.330709 0.251497 0.061538
1170 0.407080 0.164384 0.34 0.047945 0.083472 0.056338 0.021201 0.484581 0.362205 0.203593 0.323077
304 0.336283 0.363014 0.60 0.082192 0.166945 0.154930 0.296820 0.530837 0.362205 0.113772 0.123077

1279 rows × 11 columns

In [23]:
y_train
Out[23]:
952     0.8
563     0.6
150     0.6
202     0.4
638     0.8
       ... 
1119    0.4
80      0.4
1474    0.4
1170    0.6
304     0.4
Name: Signal_Strength, Length: 1279, dtype: float64
In [24]:
# Buildinhg Machine learing models
lr = LinearRegression()
lasso = Lasso()
ridge = Ridge()
svr = SVR()
knn = KNeighborsRegressor()
rf = RandomForestRegressor()
gb = GradientBoostingRegressor()

# LR
lr.fit(X_train,y_train)
lr_pred = lr.predict(X_test)
print("Linear regressiont train score = ",lr.score(X_train,y_train))

# lassso

lasso.fit(X_train,y_train)
lasso_pred = lasso.predict(X_test)
print("lasso train score = ",lasso.score(X_train,y_train))

#ridge

ridge.fit(X_train,y_train)
ridge_pred = ridge.predict(X_test)
print("ridge train score = ",ridge.score(X_train,y_train))

#SVM

svr.fit(X_train,y_train)
svr_pred = svr.predict(X_test)
print("svr train score = ",svr.score(X_train,y_train))

# knn

knn.fit(X_train,y_train)
knn_pred = knn.predict(X_test)
print("knn train score = ",knn.score(X_train,y_train))

#rf

rf.fit(X_train,y_train)
rf_pred = rf.predict(X_test)
print("random forest train score = ",rf.score(X_train,y_train))

# gb
gb.fit(X_train,y_train)
gb_pred = gb.predict(X_test)
print("graidentBoost train score = ",gb.score(X_train,y_train))
Linear regressiont train score =  0.3545899646578492
lasso train score =  0.0
ridge train score =  0.35363576550654396
svr train score =  0.49344766682183205
knn train score =  0.5391473961939335
random forest train score =  0.9287290347298892
graidentBoost train score =  0.6446850184465465
In [25]:
# scores
print("Linear regressiont test score = ",lr.score(X_test,y_test))
print("lasso test score = ",lasso.score(X_test,y_test))
print("ridge test score = ",ridge.score(X_test,y_test))
print("svr test score = ",svr.score(X_test,y_test))
print("knn test score = ",knn.score(X_test,y_test))
print("random forest test score = ",rf.score(X_test,y_test))
print("graidentBoost test score = ",gb.score(X_test,y_test))
Linear regressiont test score =  0.3760043834196448
lasso test score =  -5.5170444847929545e-05
ridge test score =  0.3764629324113328
svr test score =  0.4117812168037157
knn test score =  0.30184635873842247
random forest test score =  0.5025466432592773
graidentBoost test score =  0.41298682061231873
In [26]:
# scores
print("Linear regressiont test score = ",mean_squared_error(y_test,lr_pred))
print("lasso test score = ",mean_squared_error(y_test,lasso_pred))
print("ridge test score = ",mean_squared_error(y_test,ridge_pred))
print("svr test score = ",mean_squared_error(y_test,svr_pred))
print("knn test score = ",mean_squared_error(y_test,knn_pred))
print("random forest test score = ",mean_squared_error(y_test,rf_pred))
print("graidentBoost test score = ",mean_squared_error(y_test,gb_pred))
Linear regressiont test score =  0.016105911859829524
lasso test score =  0.025812361500903835
ridge test score =  0.01609407628046438
svr test score =  0.015182478249405357
knn test score =  0.018020000000000015
random forest test score =  0.012839737500000007
graidentBoost test score =  0.015151360484851741
In [27]:
# building Ann 
model_1 = Sequential([
    Dense(100,activation = 'relu'),
    Dense(50,activation = 'relu'),
    Dense(10,activation = 'relu'),
    Dense(1)
])

#2.compile
model_1.compile(loss = tf.keras.losses.mae,optimizer = tf.keras.optimizers.Adam(),metrics = ['mae'])

#model fitting
model_1.fit(X_train,y_train,epochs = 100)
Epoch 1/100
40/40 [==============================] - 2s 3ms/step - loss: 0.2139 - mae: 0.2139
Epoch 2/100
40/40 [==============================] - 0s 3ms/step - loss: 0.1128 - mae: 0.1128
Epoch 3/100
40/40 [==============================] - 0s 4ms/step - loss: 0.1064 - mae: 0.1064
Epoch 4/100
40/40 [==============================] - 0s 3ms/step - loss: 0.1046 - mae: 0.1046
Epoch 5/100
40/40 [==============================] - 0s 4ms/step - loss: 0.1018 - mae: 0.1018
Epoch 6/100
40/40 [==============================] - 0s 4ms/step - loss: 0.1012 - mae: 0.1012
Epoch 7/100
40/40 [==============================] - 0s 4ms/step - loss: 0.0995 - mae: 0.0995
Epoch 8/100
40/40 [==============================] - 0s 4ms/step - loss: 0.0990 - mae: 0.0990
Epoch 9/100
40/40 [==============================] - 0s 3ms/step - loss: 0.0982 - mae: 0.0982
Epoch 10/100
40/40 [==============================] - 0s 4ms/step - loss: 0.0968 - mae: 0.0968
Epoch 11/100
40/40 [==============================] - 0s 4ms/step - loss: 0.0963 - mae: 0.0963
Epoch 12/100
40/40 [==============================] - 0s 4ms/step - loss: 0.0957 - mae: 0.0957
Epoch 13/100
40/40 [==============================] - 0s 3ms/step - loss: 0.0950 - mae: 0.0950
Epoch 14/100
40/40 [==============================] - 0s 4ms/step - loss: 0.0949 - mae: 0.0949
Epoch 15/100
40/40 [==============================] - 0s 4ms/step - loss: 0.0943 - mae: 0.0943
Epoch 16/100
40/40 [==============================] - 0s 4ms/step - loss: 0.0931 - mae: 0.0931
Epoch 17/100
40/40 [==============================] - 0s 4ms/step - loss: 0.0927 - mae: 0.0927
Epoch 18/100
40/40 [==============================] - 0s 3ms/step - loss: 0.0926 - mae: 0.0926
Epoch 19/100
40/40 [==============================] - 0s 4ms/step - loss: 0.0949 - mae: 0.0949
Epoch 20/100
40/40 [==============================] - 0s 4ms/step - loss: 0.0911 - mae: 0.0911
Epoch 21/100
40/40 [==============================] - 0s 3ms/step - loss: 0.0914 - mae: 0.0914
Epoch 22/100
40/40 [==============================] - 0s 3ms/step - loss: 0.0913 - mae: 0.0913
Epoch 23/100
40/40 [==============================] - 0s 3ms/step - loss: 0.0903 - mae: 0.0903
Epoch 24/100
40/40 [==============================] - 0s 4ms/step - loss: 0.0899 - mae: 0.0899
Epoch 25/100
40/40 [==============================] - 0s 3ms/step - loss: 0.0924 - mae: 0.0924
Epoch 26/100
40/40 [==============================] - 0s 4ms/step - loss: 0.0895 - mae: 0.0895
Epoch 27/100
40/40 [==============================] - 0s 3ms/step - loss: 0.0906 - mae: 0.0906
Epoch 28/100
40/40 [==============================] - 0s 3ms/step - loss: 0.0887 - mae: 0.0887
Epoch 29/100
40/40 [==============================] - 0s 3ms/step - loss: 0.0903 - mae: 0.0903
Epoch 30/100
40/40 [==============================] - 0s 3ms/step - loss: 0.0888 - mae: 0.0888
Epoch 31/100
40/40 [==============================] - 0s 4ms/step - loss: 0.0890 - mae: 0.0890
Epoch 32/100
40/40 [==============================] - 0s 4ms/step - loss: 0.0893 - mae: 0.0893
Epoch 33/100
40/40 [==============================] - 0s 4ms/step - loss: 0.0881 - mae: 0.0881
Epoch 34/100
40/40 [==============================] - 0s 4ms/step - loss: 0.0870 - mae: 0.0870
Epoch 35/100
40/40 [==============================] - 0s 3ms/step - loss: 0.0878 - mae: 0.0878
Epoch 36/100
40/40 [==============================] - 0s 3ms/step - loss: 0.0874 - mae: 0.0874
Epoch 37/100
40/40 [==============================] - 0s 4ms/step - loss: 0.0874 - mae: 0.0874
Epoch 38/100
40/40 [==============================] - 0s 3ms/step - loss: 0.0864 - mae: 0.0864
Epoch 39/100
40/40 [==============================] - 0s 4ms/step - loss: 0.0858 - mae: 0.0858
Epoch 40/100
40/40 [==============================] - 0s 4ms/step - loss: 0.0854 - mae: 0.0854
Epoch 41/100
40/40 [==============================] - 0s 4ms/step - loss: 0.0859 - mae: 0.0859
Epoch 42/100
40/40 [==============================] - 0s 4ms/step - loss: 0.0857 - mae: 0.0857
Epoch 43/100
40/40 [==============================] - 0s 3ms/step - loss: 0.0858 - mae: 0.0858
Epoch 44/100
40/40 [==============================] - 0s 4ms/step - loss: 0.0847 - mae: 0.0847
Epoch 45/100
40/40 [==============================] - 0s 3ms/step - loss: 0.0868 - mae: 0.0868
Epoch 46/100
40/40 [==============================] - 0s 4ms/step - loss: 0.0865 - mae: 0.0865
Epoch 47/100
40/40 [==============================] - 0s 4ms/step - loss: 0.0836 - mae: 0.0836
Epoch 48/100
40/40 [==============================] - 0s 3ms/step - loss: 0.0823 - mae: 0.0823
Epoch 49/100
40/40 [==============================] - 0s 4ms/step - loss: 0.0828 - mae: 0.0828
Epoch 50/100
40/40 [==============================] - 0s 4ms/step - loss: 0.0830 - mae: 0.0830
Epoch 51/100
40/40 [==============================] - 0s 3ms/step - loss: 0.0820 - mae: 0.0820
Epoch 52/100
40/40 [==============================] - 0s 3ms/step - loss: 0.0829 - mae: 0.0829
Epoch 53/100
40/40 [==============================] - 0s 4ms/step - loss: 0.0818 - mae: 0.0818
Epoch 54/100
40/40 [==============================] - 0s 3ms/step - loss: 0.0821 - mae: 0.0821
Epoch 55/100
40/40 [==============================] - 0s 3ms/step - loss: 0.0819 - mae: 0.0819
Epoch 56/100
40/40 [==============================] - 0s 4ms/step - loss: 0.0836 - mae: 0.0836
Epoch 57/100
40/40 [==============================] - 0s 4ms/step - loss: 0.0843 - mae: 0.0843
Epoch 58/100
40/40 [==============================] - 0s 3ms/step - loss: 0.0818 - mae: 0.0818
Epoch 59/100
40/40 [==============================] - 0s 4ms/step - loss: 0.0844 - mae: 0.0844
Epoch 60/100
40/40 [==============================] - 0s 4ms/step - loss: 0.0809 - mae: 0.0809
Epoch 61/100
40/40 [==============================] - 0s 3ms/step - loss: 0.0797 - mae: 0.0797
Epoch 62/100
40/40 [==============================] - 0s 3ms/step - loss: 0.0805 - mae: 0.0805
Epoch 63/100
40/40 [==============================] - 0s 3ms/step - loss: 0.0806 - mae: 0.0806
Epoch 64/100
40/40 [==============================] - 0s 4ms/step - loss: 0.0826 - mae: 0.0826
Epoch 65/100
40/40 [==============================] - 0s 3ms/step - loss: 0.0806 - mae: 0.0806
Epoch 66/100
40/40 [==============================] - 0s 3ms/step - loss: 0.0805 - mae: 0.0805
Epoch 67/100
40/40 [==============================] - 0s 4ms/step - loss: 0.0786 - mae: 0.0786
Epoch 68/100
40/40 [==============================] - 0s 3ms/step - loss: 0.0784 - mae: 0.0784
Epoch 69/100
40/40 [==============================] - 0s 3ms/step - loss: 0.0800 - mae: 0.0800
Epoch 70/100
40/40 [==============================] - 0s 4ms/step - loss: 0.0788 - mae: 0.0788
Epoch 71/100
40/40 [==============================] - 0s 3ms/step - loss: 0.0780 - mae: 0.0780
Epoch 72/100
40/40 [==============================] - 0s 4ms/step - loss: 0.0783 - mae: 0.0783
Epoch 73/100
40/40 [==============================] - 0s 4ms/step - loss: 0.0790 - mae: 0.0790
Epoch 74/100
40/40 [==============================] - 0s 4ms/step - loss: 0.0793 - mae: 0.0793
Epoch 75/100
40/40 [==============================] - 0s 3ms/step - loss: 0.0773 - mae: 0.0773
Epoch 76/100
40/40 [==============================] - 0s 4ms/step - loss: 0.0767 - mae: 0.0767
Epoch 77/100
40/40 [==============================] - 0s 4ms/step - loss: 0.0765 - mae: 0.0765
Epoch 78/100
40/40 [==============================] - 0s 3ms/step - loss: 0.0779 - mae: 0.0779
Epoch 79/100
40/40 [==============================] - 0s 4ms/step - loss: 0.0772 - mae: 0.0772
Epoch 80/100
40/40 [==============================] - 0s 3ms/step - loss: 0.0763 - mae: 0.0763
Epoch 81/100
40/40 [==============================] - 0s 4ms/step - loss: 0.0780 - mae: 0.0780
Epoch 82/100
40/40 [==============================] - 0s 4ms/step - loss: 0.0772 - mae: 0.0772
Epoch 83/100
40/40 [==============================] - 0s 3ms/step - loss: 0.0767 - mae: 0.0767
Epoch 84/100
40/40 [==============================] - 0s 4ms/step - loss: 0.0759 - mae: 0.0759
Epoch 85/100
40/40 [==============================] - 0s 4ms/step - loss: 0.0787 - mae: 0.0787
Epoch 86/100
40/40 [==============================] - 0s 4ms/step - loss: 0.0769 - mae: 0.0769
Epoch 87/100
40/40 [==============================] - 0s 4ms/step - loss: 0.0753 - mae: 0.0753
Epoch 88/100
40/40 [==============================] - 0s 3ms/step - loss: 0.0752 - mae: 0.0752
Epoch 89/100
40/40 [==============================] - 0s 4ms/step - loss: 0.0756 - mae: 0.0756
Epoch 90/100
40/40 [==============================] - 0s 4ms/step - loss: 0.0741 - mae: 0.0741
Epoch 91/100
40/40 [==============================] - 0s 4ms/step - loss: 0.0745 - mae: 0.0745
Epoch 92/100
40/40 [==============================] - 0s 4ms/step - loss: 0.0750 - mae: 0.0750
Epoch 93/100
40/40 [==============================] - 0s 4ms/step - loss: 0.0740 - mae: 0.0740
Epoch 94/100
40/40 [==============================] - 0s 4ms/step - loss: 0.0774 - mae: 0.0774
Epoch 95/100
40/40 [==============================] - 0s 4ms/step - loss: 0.0741 - mae: 0.0741
Epoch 96/100
40/40 [==============================] - 0s 4ms/step - loss: 0.0748 - mae: 0.0748
Epoch 97/100
40/40 [==============================] - 0s 3ms/step - loss: 0.0750 - mae: 0.0750
Epoch 98/100
40/40 [==============================] - 0s 3ms/step - loss: 0.0744 - mae: 0.0744
Epoch 99/100
40/40 [==============================] - 0s 4ms/step - loss: 0.0734 - mae: 0.0734
Epoch 100/100
40/40 [==============================] - 0s 4ms/step - loss: 0.0723 - mae: 0.0723
Out[27]:
<keras.callbacks.History at 0x7f75f81323d0>
In [28]:
#check the results 
model_1.evaluate(X_test,y_test)
10/10 [==============================] - 0s 3ms/step - loss: 0.0912 - mae: 0.0912
Out[28]:
[0.0912013053894043, 0.0912013053894043]
In [29]:
y_pred1 = model_1.predict(X_test)
In [30]:
y_pred1[1]-y_test[:1]
Out[30]:
1515   -0.134352
Name: Signal_Strength, dtype: float64
In [31]:
!pip install keras_tuner
Collecting keras_tuner
  Downloading keras_tuner-1.1.0-py3-none-any.whl (98 kB)
     |████████████████████████████████| 98 kB 3.7 MB/s 
Requirement already satisfied: packaging in /usr/local/lib/python3.7/dist-packages (from keras_tuner) (21.3)
Requirement already satisfied: scipy in /usr/local/lib/python3.7/dist-packages (from keras_tuner) (1.4.1)
Collecting kt-legacy
  Downloading kt_legacy-1.0.4-py3-none-any.whl (9.6 kB)
Requirement already satisfied: tensorboard in /usr/local/lib/python3.7/dist-packages (from keras_tuner) (2.7.0)
Requirement already satisfied: numpy in /usr/local/lib/python3.7/dist-packages (from keras_tuner) (1.19.5)
Requirement already satisfied: ipython in /usr/local/lib/python3.7/dist-packages (from keras_tuner) (5.5.0)
Requirement already satisfied: requests in /usr/local/lib/python3.7/dist-packages (from keras_tuner) (2.23.0)
Requirement already satisfied: setuptools>=18.5 in /usr/local/lib/python3.7/dist-packages (from ipython->keras_tuner) (57.4.0)
Requirement already satisfied: simplegeneric>0.8 in /usr/local/lib/python3.7/dist-packages (from ipython->keras_tuner) (0.8.1)
Requirement already satisfied: prompt-toolkit<2.0.0,>=1.0.4 in /usr/local/lib/python3.7/dist-packages (from ipython->keras_tuner) (1.0.18)
Requirement already satisfied: pickleshare in /usr/local/lib/python3.7/dist-packages (from ipython->keras_tuner) (0.7.5)
Requirement already satisfied: pexpect in /usr/local/lib/python3.7/dist-packages (from ipython->keras_tuner) (4.8.0)
Requirement already satisfied: decorator in /usr/local/lib/python3.7/dist-packages (from ipython->keras_tuner) (4.4.2)
Requirement already satisfied: traitlets>=4.2 in /usr/local/lib/python3.7/dist-packages (from ipython->keras_tuner) (5.1.1)
Requirement already satisfied: pygments in /usr/local/lib/python3.7/dist-packages (from ipython->keras_tuner) (2.6.1)
Requirement already satisfied: wcwidth in /usr/local/lib/python3.7/dist-packages (from prompt-toolkit<2.0.0,>=1.0.4->ipython->keras_tuner) (0.2.5)
Requirement already satisfied: six>=1.9.0 in /usr/local/lib/python3.7/dist-packages (from prompt-toolkit<2.0.0,>=1.0.4->ipython->keras_tuner) (1.15.0)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.7/dist-packages (from packaging->keras_tuner) (3.0.7)
Requirement already satisfied: ptyprocess>=0.5 in /usr/local/lib/python3.7/dist-packages (from pexpect->ipython->keras_tuner) (0.7.0)
Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests->keras_tuner) (3.0.4)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests->keras_tuner) (1.24.3)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests->keras_tuner) (2021.10.8)
Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests->keras_tuner) (2.10)
Requirement already satisfied: tensorboard-data-server<0.7.0,>=0.6.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard->keras_tuner) (0.6.1)
Requirement already satisfied: absl-py>=0.4 in /usr/local/lib/python3.7/dist-packages (from tensorboard->keras_tuner) (1.0.0)
Requirement already satisfied: google-auth<3,>=1.6.3 in /usr/local/lib/python3.7/dist-packages (from tensorboard->keras_tuner) (1.35.0)
Requirement already satisfied: protobuf>=3.6.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard->keras_tuner) (3.17.3)
Requirement already satisfied: grpcio>=1.24.3 in /usr/local/lib/python3.7/dist-packages (from tensorboard->keras_tuner) (1.43.0)
Requirement already satisfied: tensorboard-plugin-wit>=1.6.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard->keras_tuner) (1.8.1)
Requirement already satisfied: wheel>=0.26 in /usr/local/lib/python3.7/dist-packages (from tensorboard->keras_tuner) (0.37.1)
Requirement already satisfied: google-auth-oauthlib<0.5,>=0.4.1 in /usr/local/lib/python3.7/dist-packages (from tensorboard->keras_tuner) (0.4.6)
Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.7/dist-packages (from tensorboard->keras_tuner) (3.3.6)
Requirement already satisfied: werkzeug>=0.11.15 in /usr/local/lib/python3.7/dist-packages (from tensorboard->keras_tuner) (1.0.1)
Requirement already satisfied: rsa<5,>=3.1.4 in /usr/local/lib/python3.7/dist-packages (from google-auth<3,>=1.6.3->tensorboard->keras_tuner) (4.8)
Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/local/lib/python3.7/dist-packages (from google-auth<3,>=1.6.3->tensorboard->keras_tuner) (0.2.8)
Requirement already satisfied: cachetools<5.0,>=2.0.0 in /usr/local/lib/python3.7/dist-packages (from google-auth<3,>=1.6.3->tensorboard->keras_tuner) (4.2.4)
Requirement already satisfied: requests-oauthlib>=0.7.0 in /usr/local/lib/python3.7/dist-packages (from google-auth-oauthlib<0.5,>=0.4.1->tensorboard->keras_tuner) (1.3.0)
Requirement already satisfied: importlib-metadata>=4.4 in /usr/local/lib/python3.7/dist-packages (from markdown>=2.6.8->tensorboard->keras_tuner) (4.10.1)
Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata>=4.4->markdown>=2.6.8->tensorboard->keras_tuner) (3.7.0)
Requirement already satisfied: typing-extensions>=3.6.4 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata>=4.4->markdown>=2.6.8->tensorboard->keras_tuner) (3.10.0.2)
Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in /usr/local/lib/python3.7/dist-packages (from pyasn1-modules>=0.2.1->google-auth<3,>=1.6.3->tensorboard->keras_tuner) (0.4.8)
Requirement already satisfied: oauthlib>=3.0.0 in /usr/local/lib/python3.7/dist-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<0.5,>=0.4.1->tensorboard->keras_tuner) (3.1.1)
Installing collected packages: kt-legacy, keras-tuner
Successfully installed keras-tuner-1.1.0 kt-legacy-1.0.4
In [32]:
import keras_tuner as kt
from keras_tuner import HyperModel


class ANNHyperModel(HyperModel):
    
    
    def build(self, hp):
        
        model = tf.keras.Sequential()
        # Tune the number of units in the first Dense layer
        # Choose an optimal value between 32-512
        hp_units1 = hp.Int('units1', min_value=8, max_value=512, step=4)
        hp_units2 = hp.Int('units2', min_value=16, max_value=512, step=4)
        hp_units3 = hp.Int('units3', min_value=32, max_value=512, step=4)
        model.add(Dense(units=hp_units1, activation='relu'))
        model.add(tf.keras.layers.Dense(units=hp_units2, activation='relu'))
        model.add(tf.keras.layers.Dense(units=hp_units3, activation='relu'))
        model.add(Dense(1, kernel_initializer='normal', activation='linear'))

        # Tune the learning rate for the optimizer
        # Choose an optimal value from 0.01, 0.001, or 0.0001
        hp_learning_rate = hp.Choice('learning_rate', values=[1e-2, 1e-3, 1e-4])

        model.compile(
              optimizer=tf.keras.optimizers.Adam(learning_rate=hp_learning_rate),
              loss='mse',metrics=['mse']
        )

        return model

hypermodel = ANNHyperModel()

tuner_h1 = kt.Hyperband(
    hypermodel,
    objective='mse',
    max_epochs=10,
    factor=3,
    #directory='keras_tuner_dir',
    project_name='keras_tuner_demo_hyperband'
)
In [33]:
class RegressionHyperModel(HyperModel):
    
    def build(self, hp):
        model = Sequential()
        model.add(
            Dense(
                units=hp.Int('units', 8, 512, 4, default=8),
                activation = 'relu'
            )
        )
        
        model.add(
            Dense(
                units=hp.Int('units', 16, 512, 4, default=16),
                activation = 'relu'
            )
        )
        
        model.add(
            Dense(
                units=hp.Int('units', 32, 512, 4, default=32),
                activation = 'relu'
            )
        )
        
        model.add(Dense(1))
        
        model.compile(
            optimizer='adam',loss='mse',metrics=['mse']
        )
        
        return model
    
hypermodel = RegressionHyperModel()

tuner_rs1 = kt.RandomSearch(
            hypermodel,
            objective='mse',
            seed=42,
            max_trials=10,
            executions_per_trial=2,
            directory=os.path.normpath('D:/'),
            project_name='keras_tuner_demo_random_search')
In [34]:
# hypertuing Using Keras tuner
tuner_h1.search(X_train, y_train, epochs=10)
Trial 30 Complete [00h 00m 03s]
mse: 0.01792154088616371

Best mse So Far: 0.015451465733349323
Total elapsed time: 00h 01m 14s
INFO:tensorflow:Oracle triggered exit
In [35]:
best_model_h1 = tuner_h1.get_best_models(num_models=1)[0]
loss, mse = best_model_h1.evaluate(X_test, y_test)
10/10 [==============================] - 0s 3ms/step - loss: 0.0173 - mse: 0.0173
In [36]:
best_model_h1.summary()
Model: "sequential"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 dense (Dense)               (32, 52)                  624       
                                                                 
 dense_1 (Dense)             (32, 412)                 21836     
                                                                 
 dense_2 (Dense)             (32, 504)                 208152    
                                                                 
 dense_3 (Dense)             (32, 1)                   505       
                                                                 
=================================================================
Total params: 231,117
Trainable params: 231,117
Non-trainable params: 0
_________________________________________________________________
In [37]:
# hypertuing Using Keras tuner [random_search]
tuner_rs1.search(X_train, y_train, epochs=10)
Trial 10 Complete [00h 00m 06s]
mse: 0.015380651224404573

Best mse So Far: 0.015115271788090467
Total elapsed time: 00h 01m 21s
INFO:tensorflow:Oracle triggered exit
In [38]:
best_model_rs1 = tuner_rs1.get_best_models(num_models=1)[0]
loss, mse = best_model_rs1.evaluate(X_test, y_test)
10/10 [==============================] - 0s 3ms/step - loss: 0.0157 - mse: 0.0157
In [39]:
best_model_rs1.summary()
Model: "sequential"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 dense (Dense)               (32, 260)                 3120      
                                                                 
 dense_1 (Dense)             (32, 260)                 67860     
                                                                 
 dense_2 (Dense)             (32, 260)                 67860     
                                                                 
 dense_3 (Dense)             (32, 1)                   261       
                                                                 
=================================================================
Total params: 139,101
Trainable params: 139,101
Non-trainable params: 0
_________________________________________________________________
In [40]:
y_pred_rs1 = best_model_rs1.predict(X_test)
mean_squared_error(y_test,y_pred_rs1)
Out[40]:
0.015734371348203362
In [41]:
y_pred_h1 = best_model_h1.predict(X_test)
mean_squared_error(y_test,y_pred_h1)
Out[41]:
0.017325195403155906

Handling Outliers

In [42]:
# lets handle outliers and see how rthe accuaracy improves
from scipy.stats import zscore

z_data  = zscore(data)
z_data
Out[42]:
array([[-0.52835961,  0.96187667, -1.39147228, ..., -0.57920652,
        -0.96024611, -0.78782264],
       [-0.29854743,  1.96744245, -1.39147228, ...,  0.1289504 ,
        -0.58477711, -0.78782264],
       [-0.29854743,  1.29706527, -1.18607043, ..., -0.04808883,
        -0.58477711, -0.78782264],
       ...,
       [-1.1603431 , -0.09955388, -0.72391627, ...,  0.54204194,
         0.54162988,  0.45084835],
       [-1.39015528,  0.65462046, -0.77526673, ...,  0.30598963,
        -0.20930812, -0.78782264],
       [-1.33270223, -1.21684919,  1.02199944, ...,  0.01092425,
         0.54162988,  0.45084835]])
In [43]:
# replacing the outliers with median
handle_X = data.iloc[:,:-1].copy()
for c in handle_X.columns:
    q1 = handle_X[c].quantile(.25)
    q3 = handle_X[c].quantile(.75)
    #print(c,q1,q3)
    IQR = q3-q1
    up = q3 + (IQR * 1.5)
    down = q1 - (1.5 * IQR)
    
    
    up_f = handle_X[c] >= up
    down_f = handle_X[c] <= down
    
    # replacing all the outliers with mean
    handle_X.loc[up_f,c] = handle_X[c].mean()
    handle_X.loc[up_f,c] = handle_X[c].mean()
In [44]:
handle_X
Out[44]:
Parameter 1 Parameter 2 Parameter 3 Parameter 4 Parameter 5 Parameter 6 Parameter 7 Parameter 8 Parameter 9 Parameter 10 Parameter 11
0 7.4 0.700 0.00 1.9 0.076 11.0 34.0 0.99780 3.51 0.56 9.4
1 7.8 0.880 0.00 2.6 0.098 25.0 67.0 0.99680 3.20 0.68 9.8
2 7.8 0.760 0.04 2.3 0.092 15.0 54.0 0.99700 3.26 0.65 9.8
3 11.2 0.280 0.56 1.9 0.075 17.0 60.0 0.99800 3.16 0.58 9.8
4 7.4 0.700 0.00 1.9 0.076 11.0 34.0 0.99780 3.51 0.56 9.4
... ... ... ... ... ... ... ... ... ... ... ...
1594 6.2 0.600 0.08 2.0 0.090 32.0 44.0 0.99490 3.45 0.58 10.5
1595 5.9 0.550 0.10 2.2 0.062 39.0 51.0 0.99512 3.52 0.76 11.2
1596 6.3 0.510 0.13 2.3 0.076 29.0 40.0 0.99574 3.42 0.75 11.0
1597 5.9 0.645 0.12 2.0 0.075 32.0 44.0 0.99547 3.57 0.71 10.2
1598 6.0 0.310 0.47 3.6 0.067 18.0 42.0 0.99549 3.39 0.66 11.0

1599 rows × 11 columns

In [45]:
plt.figure(figsize = (15,15))
for i in range(0,11):
    plt.subplot(3,4,i+1)
    sns.boxplot(x = handle_X.columns[i],data = handle_X)
plt.tight_layout()
plt.show()
In [46]:
X_train,X_test,y_train,y_test  = train_test_split(MinMaxScaler().fit_transform(handle_X),y,test_size=0.2,stratify=y,random_state=1)
In [47]:
#Buildinhg Machine learing models
lr = LinearRegression()
lasso = Lasso()
ridge = Ridge()
svr = SVR()
knn = KNeighborsRegressor()
rf = RandomForestRegressor()
gb = GradientBoostingRegressor()

# LR
lr.fit(X_train,y_train)
lr_pred = lr.predict(X_test)
print("Linear regressiont train score = ",lr.score(X_train,y_train))

# lassso

lasso.fit(X_train,y_train)
lasso_pred = lasso.predict(X_test)
print("lasso train score = ",lasso.score(X_train,y_train))

#ridge

ridge.fit(X_train,y_train)
ridge_pred = ridge.predict(X_test)
print("ridge train score = ",ridge.score(X_train,y_train))

#SVM

svr.fit(X_train,y_train)
svr_pred = svr.predict(X_test)
print("svr train score = ",svr.score(X_train,y_train))

# knn

knn.fit(X_train,y_train)
knn_pred = knn.predict(X_test)
print("knn train score = ",knn.score(X_train,y_train))

#rf

rf.fit(X_train,y_train)
rf_pred = rf.predict(X_test)
print("random forest train score = ",rf.score(X_train,y_train))

# gb
gb.fit(X_train,y_train)
gb_pred = gb.predict(X_test)
print("graidentBoost train score = ",gb.score(X_train,y_train))
Linear regressiont train score =  0.3559496205333986
lasso train score =  0.0
ridge train score =  0.35582545499429274
svr train score =  0.5708220472442258
knn train score =  0.52421324451264
random forest train score =  0.9258180717988063
graidentBoost train score =  0.6173724573580797
In [48]:
# scores
print("Linear regressiont test score = ",mean_squared_error(y_test,lr_pred))
print("lasso test score = ",mean_squared_error(y_test,lasso_pred))
print("ridge test score = ",mean_squared_error(y_test,ridge_pred))
print("svr test score = ",mean_squared_error(y_test,svr_pred))
print("knn test score = ",mean_squared_error(y_test,knn_pred))
print("random forest test score = ",mean_squared_error(y_test,rf_pred))
print("graidentBoost test score = ",mean_squared_error(y_test,gb_pred))
Linear regressiont test score =  0.016767451709934843
lasso test score =  0.025812361500903835
ridge test score =  0.016752651493574404
svr test score =  0.014708934736546515
knn test score =  0.01653500000000001
random forest test score =  0.013166725000000008
graidentBoost test score =  0.016120593971516446
In [49]:
model_2 = Sequential([
    Dense(100,activation = 'relu'),
    Dense(50,activation = 'relu'),
    Dense(10,activation = 'relu'),
    Dense(1)
])

#2.compile
model_2.compile(loss = tf.keras.losses.mae,optimizer = tf.keras.optimizers.Adam(),metrics = ['mae'])

#model fitting
model_2.fit(X_train,y_train,epochs = 100)
Epoch 1/100
40/40 [==============================] - 1s 4ms/step - loss: 0.1797 - mae: 0.1797
Epoch 2/100
40/40 [==============================] - 0s 4ms/step - loss: 0.1062 - mae: 0.1062
Epoch 3/100
40/40 [==============================] - 0s 4ms/step - loss: 0.1020 - mae: 0.1020
Epoch 4/100
40/40 [==============================] - 0s 3ms/step - loss: 0.1006 - mae: 0.1006
Epoch 5/100
40/40 [==============================] - 0s 3ms/step - loss: 0.0992 - mae: 0.0992
Epoch 6/100
40/40 [==============================] - 0s 4ms/step - loss: 0.0995 - mae: 0.0995
Epoch 7/100
40/40 [==============================] - 0s 4ms/step - loss: 0.0983 - mae: 0.0983
Epoch 8/100
40/40 [==============================] - 0s 4ms/step - loss: 0.0969 - mae: 0.0969
Epoch 9/100
40/40 [==============================] - 0s 3ms/step - loss: 0.0977 - mae: 0.0977
Epoch 10/100
40/40 [==============================] - 0s 4ms/step - loss: 0.0961 - mae: 0.0961
Epoch 11/100
40/40 [==============================] - 0s 3ms/step - loss: 0.0946 - mae: 0.0946
Epoch 12/100
40/40 [==============================] - 0s 4ms/step - loss: 0.0969 - mae: 0.0969
Epoch 13/100
40/40 [==============================] - 0s 4ms/step - loss: 0.0949 - mae: 0.0949
Epoch 14/100
40/40 [==============================] - 0s 4ms/step - loss: 0.0944 - mae: 0.0944
Epoch 15/100
40/40 [==============================] - 0s 4ms/step - loss: 0.0929 - mae: 0.0929
Epoch 16/100
40/40 [==============================] - 0s 4ms/step - loss: 0.0935 - mae: 0.0935
Epoch 17/100
40/40 [==============================] - 0s 3ms/step - loss: 0.0909 - mae: 0.0909
Epoch 18/100
40/40 [==============================] - 0s 3ms/step - loss: 0.0905 - mae: 0.0905
Epoch 19/100
40/40 [==============================] - 0s 3ms/step - loss: 0.0899 - mae: 0.0899
Epoch 20/100
40/40 [==============================] - 0s 3ms/step - loss: 0.0898 - mae: 0.0898
Epoch 21/100
40/40 [==============================] - 0s 4ms/step - loss: 0.0880 - mae: 0.0880
Epoch 22/100
40/40 [==============================] - 0s 3ms/step - loss: 0.0892 - mae: 0.0892
Epoch 23/100
40/40 [==============================] - 0s 4ms/step - loss: 0.0876 - mae: 0.0876
Epoch 24/100
40/40 [==============================] - 0s 4ms/step - loss: 0.0887 - mae: 0.0887
Epoch 25/100
40/40 [==============================] - 0s 3ms/step - loss: 0.0884 - mae: 0.0884
Epoch 26/100
40/40 [==============================] - 0s 3ms/step - loss: 0.0857 - mae: 0.0857
Epoch 27/100
40/40 [==============================] - 0s 4ms/step - loss: 0.0861 - mae: 0.0861
Epoch 28/100
40/40 [==============================] - 0s 3ms/step - loss: 0.0863 - mae: 0.0863
Epoch 29/100
40/40 [==============================] - 0s 3ms/step - loss: 0.0859 - mae: 0.0859
Epoch 30/100
40/40 [==============================] - 0s 4ms/step - loss: 0.0867 - mae: 0.0867
Epoch 31/100
40/40 [==============================] - 0s 4ms/step - loss: 0.0861 - mae: 0.0861
Epoch 32/100
40/40 [==============================] - 0s 3ms/step - loss: 0.0844 - mae: 0.0844
Epoch 33/100
40/40 [==============================] - 0s 3ms/step - loss: 0.0863 - mae: 0.0863
Epoch 34/100
40/40 [==============================] - 0s 4ms/step - loss: 0.0854 - mae: 0.0854
Epoch 35/100
40/40 [==============================] - 0s 3ms/step - loss: 0.0843 - mae: 0.0843
Epoch 36/100
40/40 [==============================] - 0s 4ms/step - loss: 0.0832 - mae: 0.0832
Epoch 37/100
40/40 [==============================] - 0s 4ms/step - loss: 0.0818 - mae: 0.0818
Epoch 38/100
40/40 [==============================] - 0s 3ms/step - loss: 0.0832 - mae: 0.0832
Epoch 39/100
40/40 [==============================] - 0s 4ms/step - loss: 0.0825 - mae: 0.0825
Epoch 40/100
40/40 [==============================] - 0s 4ms/step - loss: 0.0808 - mae: 0.0808
Epoch 41/100
40/40 [==============================] - 0s 3ms/step - loss: 0.0806 - mae: 0.0806
Epoch 42/100
40/40 [==============================] - 0s 4ms/step - loss: 0.0813 - mae: 0.0813
Epoch 43/100
40/40 [==============================] - 0s 4ms/step - loss: 0.0812 - mae: 0.0812
Epoch 44/100
40/40 [==============================] - 0s 3ms/step - loss: 0.0809 - mae: 0.0809
Epoch 45/100
40/40 [==============================] - 0s 3ms/step - loss: 0.0798 - mae: 0.0798
Epoch 46/100
40/40 [==============================] - 0s 4ms/step - loss: 0.0812 - mae: 0.0812
Epoch 47/100
40/40 [==============================] - 0s 3ms/step - loss: 0.0791 - mae: 0.0791
Epoch 48/100
40/40 [==============================] - 0s 3ms/step - loss: 0.0791 - mae: 0.0791
Epoch 49/100
40/40 [==============================] - 0s 3ms/step - loss: 0.0790 - mae: 0.0790
Epoch 50/100
40/40 [==============================] - 0s 4ms/step - loss: 0.0782 - mae: 0.0782
Epoch 51/100
40/40 [==============================] - 0s 4ms/step - loss: 0.0806 - mae: 0.0806
Epoch 52/100
40/40 [==============================] - 0s 4ms/step - loss: 0.0793 - mae: 0.0793
Epoch 53/100
40/40 [==============================] - 0s 4ms/step - loss: 0.0780 - mae: 0.0780
Epoch 54/100
40/40 [==============================] - 0s 4ms/step - loss: 0.0787 - mae: 0.0787
Epoch 55/100
40/40 [==============================] - 0s 4ms/step - loss: 0.0778 - mae: 0.0778
Epoch 56/100
40/40 [==============================] - 0s 3ms/step - loss: 0.0773 - mae: 0.0773
Epoch 57/100
40/40 [==============================] - 0s 4ms/step - loss: 0.0774 - mae: 0.0774
Epoch 58/100
40/40 [==============================] - 0s 4ms/step - loss: 0.0762 - mae: 0.0762
Epoch 59/100
40/40 [==============================] - 0s 3ms/step - loss: 0.0767 - mae: 0.0767
Epoch 60/100
40/40 [==============================] - 0s 4ms/step - loss: 0.0766 - mae: 0.0766
Epoch 61/100
40/40 [==============================] - 0s 3ms/step - loss: 0.0756 - mae: 0.0756
Epoch 62/100
40/40 [==============================] - 0s 4ms/step - loss: 0.0758 - mae: 0.0758
Epoch 63/100
40/40 [==============================] - 0s 3ms/step - loss: 0.0768 - mae: 0.0768
Epoch 64/100
40/40 [==============================] - 0s 4ms/step - loss: 0.0760 - mae: 0.0760
Epoch 65/100
40/40 [==============================] - 0s 4ms/step - loss: 0.0742 - mae: 0.0742
Epoch 66/100
40/40 [==============================] - 0s 3ms/step - loss: 0.0735 - mae: 0.0735
Epoch 67/100
40/40 [==============================] - 0s 3ms/step - loss: 0.0771 - mae: 0.0771
Epoch 68/100
40/40 [==============================] - 0s 3ms/step - loss: 0.0735 - mae: 0.0735
Epoch 69/100
40/40 [==============================] - 0s 4ms/step - loss: 0.0747 - mae: 0.0747
Epoch 70/100
40/40 [==============================] - 0s 4ms/step - loss: 0.0740 - mae: 0.0740
Epoch 71/100
40/40 [==============================] - 0s 4ms/step - loss: 0.0745 - mae: 0.0745
Epoch 72/100
40/40 [==============================] - 0s 3ms/step - loss: 0.0729 - mae: 0.0729
Epoch 73/100
40/40 [==============================] - 0s 4ms/step - loss: 0.0740 - mae: 0.0740
Epoch 74/100
40/40 [==============================] - 0s 4ms/step - loss: 0.0730 - mae: 0.0730
Epoch 75/100
40/40 [==============================] - 0s 4ms/step - loss: 0.0732 - mae: 0.0732
Epoch 76/100
40/40 [==============================] - 0s 4ms/step - loss: 0.0726 - mae: 0.0726
Epoch 77/100
40/40 [==============================] - 0s 4ms/step - loss: 0.0740 - mae: 0.0740
Epoch 78/100
40/40 [==============================] - 0s 4ms/step - loss: 0.0731 - mae: 0.0731
Epoch 79/100
40/40 [==============================] - 0s 4ms/step - loss: 0.0712 - mae: 0.0712
Epoch 80/100
40/40 [==============================] - 0s 3ms/step - loss: 0.0717 - mae: 0.0717
Epoch 81/100
40/40 [==============================] - 0s 4ms/step - loss: 0.0731 - mae: 0.0731
Epoch 82/100
40/40 [==============================] - 0s 4ms/step - loss: 0.0721 - mae: 0.0721
Epoch 83/100
40/40 [==============================] - 0s 4ms/step - loss: 0.0711 - mae: 0.0711
Epoch 84/100
40/40 [==============================] - 0s 4ms/step - loss: 0.0705 - mae: 0.0705
Epoch 85/100
40/40 [==============================] - 0s 3ms/step - loss: 0.0727 - mae: 0.0727
Epoch 86/100
40/40 [==============================] - 0s 3ms/step - loss: 0.0697 - mae: 0.0697
Epoch 87/100
40/40 [==============================] - 0s 3ms/step - loss: 0.0716 - mae: 0.0716
Epoch 88/100
40/40 [==============================] - 0s 4ms/step - loss: 0.0712 - mae: 0.0712
Epoch 89/100
40/40 [==============================] - 0s 3ms/step - loss: 0.0754 - mae: 0.0754
Epoch 90/100
40/40 [==============================] - 0s 4ms/step - loss: 0.0703 - mae: 0.0703
Epoch 91/100
40/40 [==============================] - 0s 4ms/step - loss: 0.0686 - mae: 0.0686
Epoch 92/100
40/40 [==============================] - 0s 4ms/step - loss: 0.0704 - mae: 0.0704
Epoch 93/100
40/40 [==============================] - 0s 3ms/step - loss: 0.0698 - mae: 0.0698
Epoch 94/100
40/40 [==============================] - 0s 4ms/step - loss: 0.0699 - mae: 0.0699
Epoch 95/100
40/40 [==============================] - 0s 4ms/step - loss: 0.0700 - mae: 0.0700
Epoch 96/100
40/40 [==============================] - 0s 4ms/step - loss: 0.0690 - mae: 0.0690
Epoch 97/100
40/40 [==============================] - 0s 4ms/step - loss: 0.0691 - mae: 0.0691
Epoch 98/100
40/40 [==============================] - 0s 4ms/step - loss: 0.0683 - mae: 0.0683
Epoch 99/100
40/40 [==============================] - 0s 3ms/step - loss: 0.0706 - mae: 0.0706
Epoch 100/100
40/40 [==============================] - 0s 3ms/step - loss: 0.0688 - mae: 0.0688
Out[49]:
<keras.callbacks.History at 0x7f75f80a6610>
In [50]:
#check the results 
model_2.evaluate(X_test,y_test)
10/10 [==============================] - 0s 3ms/step - loss: 0.0942 - mae: 0.0942
Out[50]:
[0.09421882033348083, 0.09421882033348083]
In [51]:
# hypertuing Using Keras tuner
hypermodel = ANNHyperModel()

tuner_h2 = kt.Hyperband(
    hypermodel,
    objective='mse',
    max_epochs=10,
    factor=3,
    #directory='keras_tuner_dir',
    project_name='keras_tuner_demo_h2'
)

tuner_h2.search(X_train, y_train, epochs=10)
Trial 30 Complete [00h 00m 02s]
mse: 0.019674809649586678

Best mse So Far: 0.016465649008750916
Total elapsed time: 00h 00m 48s
INFO:tensorflow:Oracle triggered exit
In [52]:
best_model_h2 = tuner_h2.get_best_models(num_models=1)[0]
loss, mse = best_model_h2.evaluate(X_test, y_test)
10/10 [==============================] - 0s 3ms/step - loss: 0.0163 - mse: 0.0163
In [53]:
best_model_h2.summary()
Model: "sequential"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 dense (Dense)               (32, 160)                 1920      
                                                                 
 dense_1 (Dense)             (32, 72)                  11592     
                                                                 
 dense_2 (Dense)             (32, 188)                 13724     
                                                                 
 dense_3 (Dense)             (32, 1)                   189       
                                                                 
=================================================================
Total params: 27,425
Trainable params: 27,425
Non-trainable params: 0
_________________________________________________________________
In [54]:
# random_search 
hypermodel =RegressionHyperModel()

tuner_rs2 = kt.RandomSearch(
            hypermodel,
            objective='mse',
            seed=42,
            max_trials=10,
            executions_per_trial=2,
            directory=os.path.normpath('D:/'),
            project_name='keras_tuner_demo__rs2')

tuner_rs2.search(X_train, y_train, epochs=10)
Trial 10 Complete [00h 00m 05s]
mse: 0.015621459111571312

Best mse So Far: 0.01510324701666832
Total elapsed time: 00h 01m 02s
INFO:tensorflow:Oracle triggered exit
In [55]:
best_model_rs2 = tuner_rs2.get_best_models(num_models=1)[0]
loss, mse = best_model_rs2.evaluate(X_test, y_test)
10/10 [==============================] - 0s 3ms/step - loss: 0.0159 - mse: 0.0159
In [56]:
best_model_rs2.summary()
Model: "sequential"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 dense (Dense)               (32, 212)                 2544      
                                                                 
 dense_1 (Dense)             (32, 212)                 45156     
                                                                 
 dense_2 (Dense)             (32, 212)                 45156     
                                                                 
 dense_3 (Dense)             (32, 1)                   213       
                                                                 
=================================================================
Total params: 93,069
Trainable params: 93,069
Non-trainable params: 0
_________________________________________________________________
In [57]:
y_pred_rs2 = best_model_rs2.predict(X_test)
mean_squared_error(y_test,y_pred_rs2)
Out[57]:
0.01590262432860443

Conclusion

  • we have used various machine learning Modles and simple Ann and hypertuned Ann
  • For Hypertunning we used two different methods hyperband and Random search
  • we have tried the whole algorithms using data with outliers and data where outliers are handled
  • When outliers are present then best model fromhyperband performs well
  • when outliers are handled then best model from random search is performing better
  • Overall the best from hyperband is performing better with data where outliers are present
In [53]:
# saving the better model for future use 

# 1. When there are outliers 
best_model_h1.save("best_model_with_outliers")

# 2. when there is no outliers 
best_model_rs2.save("best_model_without_outliers")
INFO:tensorflow:Assets written to: best_model_with_outliers/assets
INFO:tensorflow:Assets written to: best_model_without_outliers/assets

*** END PART 1 **

Part - 2

•DOMAIN: Autonomous Vehicles

•BUSINESS CONTEXT: A Recognising multi-digit numbers in photographs captured at street level is an important component of modern-day map making. A classic example of a corpus of such street-level photographs is Google’s Street View imagery composed of hundreds of millions of geo-located 360-degree panoramic images. The ability to automatically transcribe an address number from a geo-located patch of pixels and associate the transcribed number with a known street address helps pinpoint, with a high degree of accuracy, the location of the building it represents. More broadly, recognising numbers in photographs is a problem of interest to the optical character recognition community. While OCR on constrained domains like document processing is well studied, arbitrary multi-character text recognition in photographs is still highly challenging. This difficulty arises due to the wide variability in the visual appearance of text in the wild on account of a large range of fonts, colours, styles, orientations, and character arrangements. The recognition problem is further complicated by environmental factors such as lighting, shadows, specularity, and occlusions as well as by image acquisition factors such as resolution, motion, and focus blurs. In this project, we will use the dataset with images centred around a single digit (many of the images do contain some distractors at the sides). Although we are taking a sample of the data which is simpler, it is more complex than MNIST because of the distractors.

•DATA DESCRIPTION: The SVHN is a real-world image dataset for developing machine learning and object recognition algorithms with the minimal requirement on data formatting but comes from a significantly harder, unsolved, real-world problem (recognising digits and numbers in natural scene images). SVHN is obtained from house numbers in Google Street View images.Where the labels for each of this image are the prominent number in that image i.e. 2,6,7 and 4 respectively.The dataset has been provided in the form of h5py files.

•PROJECT OBJECTIVE: We will build a digit classifier on the SVHN (Street View Housing Number) dataset

In [58]:
# Getting the packages
import h5py
from keras.utils.np_utils import to_categorical
from keras.layers import BatchNormalization, Dropout
from sklearn.neighbors import KNeighborsClassifier
from sklearn.metrics import confusion_matrix,accuracy_score
from sklearn.metrics import classification_report
In [59]:
from google.colab import drive
drive.mount('/content/drive')
Drive already mounted at /content/drive; to attempt to forcibly remount, call drive.mount("/content/drive", force_remount=True).
In [60]:
# getting the dataset
data1=h5py.File("/content/drive/MyDrive/Part - 4 - Autonomous_Vehicles_SVHN_single_grey1.h5",'r')


#Load the training, testing, and validation data
X_train=data1['X_train']
X_test=data1['X_test']
X_val=data1['X_val']
y_train=data1['y_train']
y_test=data1['y_test']
y_val=data1['y_val']
In [61]:
!pwd
/content
In [62]:
X_train[0]
Out[62]:
array([[ 33.0704,  30.2601,  26.852 , ...,  71.4471,  58.2204,  42.9939],
       [ 25.2283,  25.5533,  29.9765, ..., 113.0209, 103.3639,  84.2949],
       [ 26.2775,  22.6137,  40.4763, ..., 113.3028, 121.775 , 115.4228],
       ...,
       [ 28.5502,  36.212 ,  45.0801, ...,  24.1359,  25.0927,  26.0603],
       [ 38.4352,  26.4733,  23.2717, ...,  28.1094,  29.4683,  30.0661],
       [ 50.2984,  26.0773,  24.0389, ...,  49.6682,  50.853 ,  53.0377]],
      dtype=float32)
In [63]:
# Printing the shape and data type of training, testing, and validation data
print("Training data X -- Shape :", X_train.shape,"and Data Type : ", X_train.dtype)
print("Testing data X -- Shape :", X_test.shape,"and Data Type : ", X_test.dtype)
print("Validation data X -- Shape :", X_val.shape,"and Data Type : ", X_val.dtype)
print("Training data y -- Shape :", y_train.shape,"and Data Type : ", y_train.dtype)
print("Testing data y -- Shape :", y_test.shape,"and Data Type : ", y_test.dtype)
print("Validation data y -- Shape :", y_val.shape,"and Data Type : ", y_val.dtype)
Training data X -- Shape : (42000, 32, 32) and Data Type :  float32
Testing data X -- Shape : (18000, 32, 32) and Data Type :  float32
Validation data X -- Shape : (60000, 32, 32) and Data Type :  float32
Training data y -- Shape : (42000,) and Data Type :  uint8
Testing data y -- Shape : (18000,) and Data Type :  uint8
Validation data y -- Shape : (60000,) and Data Type :  uint8
In [64]:
fig=plt.figure(figsize=(8,8))
columns=10
rows=10
for i in range(1, columns*rows+1):
    img=X_test[i]
    fig.add_subplot(rows,columns,i)
    plt.imshow(img,cmap='gray')
plt.show()
In [66]:
# show the number in the dataset
plt.imshow(X_train[0],cmap='gray')    
plt.show()
print('Label: ', y_train[0])
Label:  2
In [67]:
# show the number in the dataset
plt.imshow(X_test[0],cmap='gray')    
plt.show()
print('Label: ', y_test[0])
Label:  1
In [68]:
# visualizing the first 10 images in the dataset and their labels
%matplotlib inline
import matplotlib.pyplot as plt
plt.figure(figsize=(10, 1))
for i in range(10):
    plt.subplot(1, 10, i+1)
    plt.imshow(X_train[i].reshape(32,32),cmap='gray')
    plt.axis('off')
plt.show()
print('label for each of the above image: %s' % (y_train[0:10]))
label for each of the above image: [2 6 7 4 4 0 3 0 7 3]
In [69]:
#Reshape data from 2D to 1D -> 32X32 to 1024
X_train = np.asarray(X_train).reshape(42000,1024)
X_test = np.asarray(X_test).reshape(18000,1024)
X_val = np.asarray(X_val).reshape(60000,1024)

X_train=X_train/255
X_test=X_test/255
X_val= X_val/255
In [70]:
# Initializing the value of k and finding the accuracies on validation data
k_vals = range(1, 13, 2)
accuracies = []

for k in range(1, 13, 2):
    
    knn = KNeighborsClassifier(n_neighbors=k)
    knn.fit(X_train, y_train)
    score = knn.score(X_val, y_val)
    print("k value=%d, accuracy score=%.2f%%" % (k, score * 100))
    accuracies.append(score)

# finding the value of k which has the largest accuracy
i = int(np.argmax(accuracies))
print("k=%d value has highest accuracy of %.2f%% on validation data" % (k_vals[i],accuracies[i] * 100))
k value=1, accuracy score=83.78%
k value=3, accuracy score=62.83%
k value=5, accuracy score=62.42%
k value=7, accuracy score=61.42%
k value=9, accuracy score=60.59%
k value=11, accuracy score=60.32%
k=1 value has highest accuracy of 83.78% on validation data
In [71]:
# Retraining the model using the best k value and predict the labels on test data

knn = KNeighborsClassifier(n_neighbors=k_vals[i])
knn.fit(X_train, y_train)
predictions = knn.predict(X_test)
In [72]:
# show a final classification report demonstrating the accuracy of the classifier
print("EVALUATION ON TESTING DATA")
print(confusion_matrix(y_test,predictions))
print(classification_report(y_test, predictions))
EVALUATION ON TESTING DATA
[[ 935   43   51   64   71   65  185   40  170  190]
 [  51 1034  109  122  123   69   68  119   67   66]
 [  63  135  872  131   82   74   65  168   79  134]
 [  67  166  127  610   88  229   82   80  149  121]
 [  85  195   51   71 1039   55  111   35   98   72]
 [  94  103   71  220   54  588  203   48  205  182]
 [ 225   74   51   72  102  154  711   33  295  115]
 [  65  173  146  100   34   38   48 1071   53   80]
 [ 150   62   59  130   86  153  267   44  675  186]
 [ 238   87   89  115   60  113  115   65  192  730]]
              precision    recall  f1-score   support

           0       0.47      0.52      0.49      1814
           1       0.50      0.57      0.53      1828
           2       0.54      0.48      0.51      1803
           3       0.37      0.35      0.36      1719
           4       0.60      0.57      0.59      1812
           5       0.38      0.33      0.36      1768
           6       0.38      0.39      0.39      1832
           7       0.63      0.59      0.61      1808
           8       0.34      0.37      0.36      1812
           9       0.39      0.40      0.40      1804

    accuracy                           0.46     18000
   macro avg       0.46      0.46      0.46     18000
weighted avg       0.46      0.46      0.46     18000

In [73]:
plt.figure(figsize=(2,2))
plt.imshow(X_test[59].reshape(32,32))
plt.show()
image = X_test[15]
print(knn.predict(image.reshape(1, -1)))
[9]

5. Implementing Deep Neural Network

In [74]:
#Reshape data from 2D to 1D -> 32X32 to 1024
X_train = np.asarray(X_train).reshape(42000,1024)
X_test = np.asarray(X_test).reshape(18000,1024)
X_val = np.asarray(X_val).reshape(60000,1024)

Converting output label to multiple values

In [75]:
y_train = tf.keras.utils.to_categorical(y_train, num_classes=10)
In [76]:
y_test = tf.keras.utils.to_categorical(y_test, num_classes=10)
In [77]:
y_val = tf.keras.utils.to_categorical(y_val, num_classes=10)
In [78]:
print(X_train.shape, X_test.shape,X_val.shape, y_train.shape, y_test.shape,y_val.shape)
(42000, 1024) (18000, 1024) (60000, 1024) (42000, 10) (18000, 10) (60000, 10)
In [79]:
y_train = y_train.astype('int')
y_test = y_test.astype('int')
y_val = y_val.astype('int')
In [80]:
X_train[0]/255
Out[80]:
array([0.00050858, 0.00046536, 0.00041295, ..., 0.00076383, 0.00078205,
       0.00081565], dtype=float32)
In [81]:
np.isnan(X_train).sum()
Out[81]:
0
In [82]:
np.isnan(y_train).sum()
Out[82]:
0
In [83]:
np.isnan(X_val).sum()
Out[83]:
0
In [84]:
np.isnan(y_val).sum()
Out[84]:
0

Implementing Neural Network

In [85]:
##Initialize the Artificial Neural Network Classifier
keras_model = Sequential()
In [86]:
# Input Layer
#Adding Input layer and activation functions ReLU
keras_model.add(Dense(512,activation='relu'))
#Adding Activation function
# kernel_initializer='he_normal'
#keras_model.add(Activation('relu'))

#Hidden Layer 1
#Adding first Hidden layer
keras_model.add(Dense(256,activation='relu'))
#Adding Activation function
#keras_model.add(Activation('relu'))

#Hidden Layer 2
#Adding second Hidden layer
keras_model.add(Dense(128,activation='relu'))
#Adding Activation function
#keras_model.add(Activation('relu'))

#Hidden Layer 3
#Adding third Hidden layer
keras_model.add(Dense(64,activation='relu'))
#Adding Activation function
#keras_model.add(Activation('relu'))

#Hidden Layer 4
#Adding fourth Hidden layer
keras_model.add(Dense(32,activation='relu'))
#Adding Activation function
#keras_model.add(Activation('relu'))

# Output Layer
#Adding output layer which is of 10 nodes (digits)
keras_model.add(Dense(10,activation='softmax'))
#Adding Activation function
# Here, we are using softmax function because we have multiclass classsification
#keras_model.add(Activation('softmax'))

Using sgd optimizer and Categorical Crossentropy as loss function and Accuracy as the metric to monitor

In [87]:
# compiling the ANN classifier
keras_model.compile(loss=tf.keras.losses.CategoricalCrossentropy(),optimizer = optimizers.Adam(),metrics = ['accuracy'])
In [88]:
# Fitting the ANN to the Training data
history = keras_model.fit(X_train/255, y_train, validation_data=(X_val/255,y_val),batch_size = 128, epochs = 10, verbose = 1)
Epoch 1/10
329/329 [==============================] - 4s 10ms/step - loss: 2.3028 - accuracy: 0.0981 - val_loss: 2.3026 - val_accuracy: 0.1000
Epoch 2/10
329/329 [==============================] - 4s 13ms/step - loss: 2.3027 - accuracy: 0.0992 - val_loss: 2.3026 - val_accuracy: 0.1000
Epoch 3/10
329/329 [==============================] - 3s 10ms/step - loss: 2.3027 - accuracy: 0.1005 - val_loss: 2.3026 - val_accuracy: 0.1000
Epoch 4/10
329/329 [==============================] - 4s 13ms/step - loss: 2.3027 - accuracy: 0.0993 - val_loss: 2.3026 - val_accuracy: 0.1000
Epoch 5/10
329/329 [==============================] - 4s 13ms/step - loss: 2.3027 - accuracy: 0.1019 - val_loss: 2.3026 - val_accuracy: 0.1000
Epoch 6/10
329/329 [==============================] - 4s 13ms/step - loss: 2.3027 - accuracy: 0.1019 - val_loss: 2.3026 - val_accuracy: 0.1000
Epoch 7/10
329/329 [==============================] - 3s 9ms/step - loss: 2.3027 - accuracy: 0.0991 - val_loss: 2.3026 - val_accuracy: 0.1000
Epoch 8/10
329/329 [==============================] - 3s 9ms/step - loss: 2.3027 - accuracy: 0.1009 - val_loss: 2.3026 - val_accuracy: 0.1000
Epoch 9/10
329/329 [==============================] - 3s 9ms/step - loss: 2.3027 - accuracy: 0.1019 - val_loss: 2.3026 - val_accuracy: 0.1000
Epoch 10/10
329/329 [==============================] - 4s 13ms/step - loss: 2.3027 - accuracy: 0.1019 - val_loss: 2.3026 - val_accuracy: 0.1000
In [89]:
keras_model.summary()
Model: "sequential_1"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 dense_4 (Dense)             (None, 512)               524800    
                                                                 
 dense_5 (Dense)             (None, 256)               131328    
                                                                 
 dense_6 (Dense)             (None, 128)               32896     
                                                                 
 dense_7 (Dense)             (None, 64)                8256      
                                                                 
 dense_8 (Dense)             (None, 32)                2080      
                                                                 
 dense_9 (Dense)             (None, 10)                330       
                                                                 
=================================================================
Total params: 699,690
Trainable params: 699,690
Non-trainable params: 0
_________________________________________________________________

Using adam optimizer and Categorical Crossentropy as loss function and Accuracy as the metric to monitor

In [90]:
# compiling the ANN classifier
keras_model.compile(optimizer = 'adam', loss = 'categorical_crossentropy', metrics = ['accuracy'])
In [91]:
# Fitting the ANN to the Training data
history = keras_model.fit(X_train, y_train, validation_data=(X_val,y_val),batch_size = 200, epochs = 10, verbose = 1)
Epoch 1/10
210/210 [==============================] - 3s 12ms/step - loss: 2.2507 - accuracy: 0.1365 - val_loss: 2.0663 - val_accuracy: 0.2401
Epoch 2/10
210/210 [==============================] - 2s 11ms/step - loss: 1.7982 - accuracy: 0.3407 - val_loss: 1.5715 - val_accuracy: 0.4449
Epoch 3/10
210/210 [==============================] - 3s 14ms/step - loss: 1.4882 - accuracy: 0.4857 - val_loss: 1.3983 - val_accuracy: 0.5309
Epoch 4/10
210/210 [==============================] - 3s 16ms/step - loss: 1.2878 - accuracy: 0.5763 - val_loss: 1.1921 - val_accuracy: 0.6155
Epoch 5/10
210/210 [==============================] - 3s 13ms/step - loss: 1.1424 - accuracy: 0.6345 - val_loss: 1.0821 - val_accuracy: 0.6575
Epoch 6/10
210/210 [==============================] - 3s 16ms/step - loss: 1.0654 - accuracy: 0.6616 - val_loss: 1.0116 - val_accuracy: 0.6850
Epoch 7/10
210/210 [==============================] - 3s 14ms/step - loss: 1.0336 - accuracy: 0.6735 - val_loss: 0.9927 - val_accuracy: 0.6873
Epoch 8/10
210/210 [==============================] - 4s 21ms/step - loss: 0.9775 - accuracy: 0.6933 - val_loss: 0.9579 - val_accuracy: 0.7008
Epoch 9/10
210/210 [==============================] - 2s 11ms/step - loss: 0.9385 - accuracy: 0.7043 - val_loss: 0.9441 - val_accuracy: 0.6981
Epoch 10/10
210/210 [==============================] - 2s 11ms/step - loss: 0.9002 - accuracy: 0.7170 - val_loss: 0.8678 - val_accuracy: 0.7287
  • The accuracy of the model on val data is very low either using adam or sgd.

Now, try to change the learning rate in SGD optimizer

As, we are taking learning rate small means model will learn slowly. So, we have to take more number of epochs.

In [92]:
#Setting the learning rate  and decay in sgd
sgd = optimizers.SGD(learning_rate = 0.0001)
# compiling the ANN classifier
keras_model.compile(optimizer = sgd, loss = 'categorical_crossentropy', metrics = ['accuracy'])
# Fitting the ANN to the Training data
history = keras_model.fit(X_train, y_train, validation_data=(X_val,y_val),batch_size = 200, epochs = 100, verbose = 1)
Epoch 1/100
210/210 [==============================] - 3s 12ms/step - loss: 0.8272 - accuracy: 0.7420 - val_loss: 0.8293 - val_accuracy: 0.7428
Epoch 2/100
210/210 [==============================] - 2s 9ms/step - loss: 0.8141 - accuracy: 0.7466 - val_loss: 0.8250 - val_accuracy: 0.7446
Epoch 3/100
210/210 [==============================] - 2s 9ms/step - loss: 0.8109 - accuracy: 0.7472 - val_loss: 0.8229 - val_accuracy: 0.7458
Epoch 4/100
210/210 [==============================] - 2s 9ms/step - loss: 0.8090 - accuracy: 0.7483 - val_loss: 0.8214 - val_accuracy: 0.7459
Epoch 5/100
210/210 [==============================] - 2s 11ms/step - loss: 0.8077 - accuracy: 0.7490 - val_loss: 0.8203 - val_accuracy: 0.7462
Epoch 6/100
210/210 [==============================] - 2s 9ms/step - loss: 0.8065 - accuracy: 0.7492 - val_loss: 0.8195 - val_accuracy: 0.7461
Epoch 7/100
210/210 [==============================] - 2s 9ms/step - loss: 0.8058 - accuracy: 0.7489 - val_loss: 0.8188 - val_accuracy: 0.7465
Epoch 8/100
210/210 [==============================] - 2s 9ms/step - loss: 0.8050 - accuracy: 0.7496 - val_loss: 0.8182 - val_accuracy: 0.7467
Epoch 9/100
210/210 [==============================] - 2s 11ms/step - loss: 0.8044 - accuracy: 0.7500 - val_loss: 0.8177 - val_accuracy: 0.7468
Epoch 10/100
210/210 [==============================] - 2s 8ms/step - loss: 0.8038 - accuracy: 0.7503 - val_loss: 0.8172 - val_accuracy: 0.7470
Epoch 11/100
210/210 [==============================] - 2s 9ms/step - loss: 0.8033 - accuracy: 0.7504 - val_loss: 0.8167 - val_accuracy: 0.7471
Epoch 12/100
210/210 [==============================] - 2s 11ms/step - loss: 0.8029 - accuracy: 0.7506 - val_loss: 0.8163 - val_accuracy: 0.7477
Epoch 13/100
210/210 [==============================] - 2s 11ms/step - loss: 0.8024 - accuracy: 0.7508 - val_loss: 0.8160 - val_accuracy: 0.7477
Epoch 14/100
210/210 [==============================] - 2s 11ms/step - loss: 0.8020 - accuracy: 0.7511 - val_loss: 0.8157 - val_accuracy: 0.7475
Epoch 15/100
210/210 [==============================] - 2s 11ms/step - loss: 0.8016 - accuracy: 0.7510 - val_loss: 0.8153 - val_accuracy: 0.7475
Epoch 16/100
210/210 [==============================] - 2s 11ms/step - loss: 0.8013 - accuracy: 0.7514 - val_loss: 0.8150 - val_accuracy: 0.7478
Epoch 17/100
210/210 [==============================] - 2s 11ms/step - loss: 0.8010 - accuracy: 0.7515 - val_loss: 0.8147 - val_accuracy: 0.7479
Epoch 18/100
210/210 [==============================] - 2s 9ms/step - loss: 0.8006 - accuracy: 0.7516 - val_loss: 0.8144 - val_accuracy: 0.7479
Epoch 19/100
210/210 [==============================] - 2s 11ms/step - loss: 0.8003 - accuracy: 0.7518 - val_loss: 0.8141 - val_accuracy: 0.7479
Epoch 20/100
210/210 [==============================] - 2s 9ms/step - loss: 0.8000 - accuracy: 0.7520 - val_loss: 0.8138 - val_accuracy: 0.7479
Epoch 21/100
210/210 [==============================] - 2s 9ms/step - loss: 0.7997 - accuracy: 0.7516 - val_loss: 0.8136 - val_accuracy: 0.7482
Epoch 22/100
210/210 [==============================] - 2s 11ms/step - loss: 0.7995 - accuracy: 0.7519 - val_loss: 0.8133 - val_accuracy: 0.7480
Epoch 23/100
210/210 [==============================] - 2s 11ms/step - loss: 0.7992 - accuracy: 0.7518 - val_loss: 0.8131 - val_accuracy: 0.7481
Epoch 24/100
210/210 [==============================] - 2s 9ms/step - loss: 0.7990 - accuracy: 0.7522 - val_loss: 0.8129 - val_accuracy: 0.7481
Epoch 25/100
210/210 [==============================] - 2s 9ms/step - loss: 0.7987 - accuracy: 0.7516 - val_loss: 0.8127 - val_accuracy: 0.7481
Epoch 26/100
210/210 [==============================] - 2s 9ms/step - loss: 0.7985 - accuracy: 0.7522 - val_loss: 0.8124 - val_accuracy: 0.7482
Epoch 27/100
210/210 [==============================] - 2s 11ms/step - loss: 0.7982 - accuracy: 0.7522 - val_loss: 0.8122 - val_accuracy: 0.7484
Epoch 28/100
210/210 [==============================] - 2s 8ms/step - loss: 0.7980 - accuracy: 0.7520 - val_loss: 0.8120 - val_accuracy: 0.7483
Epoch 29/100
210/210 [==============================] - 2s 9ms/step - loss: 0.7978 - accuracy: 0.7520 - val_loss: 0.8118 - val_accuracy: 0.7484
Epoch 30/100
210/210 [==============================] - 2s 9ms/step - loss: 0.7976 - accuracy: 0.7522 - val_loss: 0.8116 - val_accuracy: 0.7486
Epoch 31/100
210/210 [==============================] - 2s 9ms/step - loss: 0.7974 - accuracy: 0.7525 - val_loss: 0.8114 - val_accuracy: 0.7485
Epoch 32/100
210/210 [==============================] - 2s 9ms/step - loss: 0.7972 - accuracy: 0.7524 - val_loss: 0.8112 - val_accuracy: 0.7485
Epoch 33/100
210/210 [==============================] - 2s 9ms/step - loss: 0.7970 - accuracy: 0.7526 - val_loss: 0.8110 - val_accuracy: 0.7488
Epoch 34/100
210/210 [==============================] - 2s 9ms/step - loss: 0.7968 - accuracy: 0.7526 - val_loss: 0.8109 - val_accuracy: 0.7488
Epoch 35/100
210/210 [==============================] - 2s 11ms/step - loss: 0.7966 - accuracy: 0.7525 - val_loss: 0.8107 - val_accuracy: 0.7490
Epoch 36/100
210/210 [==============================] - 2s 11ms/step - loss: 0.7964 - accuracy: 0.7527 - val_loss: 0.8106 - val_accuracy: 0.7489
Epoch 37/100
210/210 [==============================] - 2s 11ms/step - loss: 0.7962 - accuracy: 0.7526 - val_loss: 0.8104 - val_accuracy: 0.7489
Epoch 38/100
210/210 [==============================] - 2s 11ms/step - loss: 0.7960 - accuracy: 0.7529 - val_loss: 0.8103 - val_accuracy: 0.7488
Epoch 39/100
210/210 [==============================] - 2s 9ms/step - loss: 0.7959 - accuracy: 0.7530 - val_loss: 0.8101 - val_accuracy: 0.7489
Epoch 40/100
210/210 [==============================] - 2s 9ms/step - loss: 0.7957 - accuracy: 0.7528 - val_loss: 0.8099 - val_accuracy: 0.7488
Epoch 41/100
210/210 [==============================] - 2s 11ms/step - loss: 0.7955 - accuracy: 0.7527 - val_loss: 0.8098 - val_accuracy: 0.7490
Epoch 42/100
210/210 [==============================] - 2s 9ms/step - loss: 0.7954 - accuracy: 0.7530 - val_loss: 0.8097 - val_accuracy: 0.7492
Epoch 43/100
210/210 [==============================] - 2s 11ms/step - loss: 0.7952 - accuracy: 0.7534 - val_loss: 0.8095 - val_accuracy: 0.7492
Epoch 44/100
210/210 [==============================] - 2s 9ms/step - loss: 0.7951 - accuracy: 0.7531 - val_loss: 0.8094 - val_accuracy: 0.7492
Epoch 45/100
210/210 [==============================] - 2s 9ms/step - loss: 0.7949 - accuracy: 0.7534 - val_loss: 0.8092 - val_accuracy: 0.7490
Epoch 46/100
210/210 [==============================] - 2s 11ms/step - loss: 0.7948 - accuracy: 0.7531 - val_loss: 0.8091 - val_accuracy: 0.7491
Epoch 47/100
210/210 [==============================] - 2s 9ms/step - loss: 0.7946 - accuracy: 0.7529 - val_loss: 0.8090 - val_accuracy: 0.7492
Epoch 48/100
210/210 [==============================] - 2s 9ms/step - loss: 0.7945 - accuracy: 0.7533 - val_loss: 0.8089 - val_accuracy: 0.7494
Epoch 49/100
210/210 [==============================] - 2s 11ms/step - loss: 0.7943 - accuracy: 0.7530 - val_loss: 0.8088 - val_accuracy: 0.7493
Epoch 50/100
210/210 [==============================] - 2s 11ms/step - loss: 0.7942 - accuracy: 0.7530 - val_loss: 0.8086 - val_accuracy: 0.7494
Epoch 51/100
210/210 [==============================] - 3s 16ms/step - loss: 0.7941 - accuracy: 0.7534 - val_loss: 0.8084 - val_accuracy: 0.7494
Epoch 52/100
210/210 [==============================] - 3s 13ms/step - loss: 0.7939 - accuracy: 0.7530 - val_loss: 0.8083 - val_accuracy: 0.7494
Epoch 53/100
210/210 [==============================] - 2s 9ms/step - loss: 0.7938 - accuracy: 0.7536 - val_loss: 0.8082 - val_accuracy: 0.7494
Epoch 54/100
210/210 [==============================] - 2s 9ms/step - loss: 0.7937 - accuracy: 0.7534 - val_loss: 0.8081 - val_accuracy: 0.7496
Epoch 55/100
210/210 [==============================] - 2s 11ms/step - loss: 0.7935 - accuracy: 0.7536 - val_loss: 0.8080 - val_accuracy: 0.7495
Epoch 56/100
210/210 [==============================] - 2s 11ms/step - loss: 0.7934 - accuracy: 0.7535 - val_loss: 0.8079 - val_accuracy: 0.7496
Epoch 57/100
210/210 [==============================] - 2s 9ms/step - loss: 0.7933 - accuracy: 0.7536 - val_loss: 0.8077 - val_accuracy: 0.7497
Epoch 58/100
210/210 [==============================] - 2s 11ms/step - loss: 0.7932 - accuracy: 0.7535 - val_loss: 0.8076 - val_accuracy: 0.7497
Epoch 59/100
210/210 [==============================] - 2s 11ms/step - loss: 0.7930 - accuracy: 0.7537 - val_loss: 0.8075 - val_accuracy: 0.7495
Epoch 60/100
210/210 [==============================] - 2s 11ms/step - loss: 0.7929 - accuracy: 0.7540 - val_loss: 0.8074 - val_accuracy: 0.7496
Epoch 61/100
210/210 [==============================] - 2s 11ms/step - loss: 0.7928 - accuracy: 0.7538 - val_loss: 0.8073 - val_accuracy: 0.7497
Epoch 62/100
210/210 [==============================] - 2s 9ms/step - loss: 0.7927 - accuracy: 0.7540 - val_loss: 0.8072 - val_accuracy: 0.7496
Epoch 63/100
210/210 [==============================] - 2s 8ms/step - loss: 0.7925 - accuracy: 0.7540 - val_loss: 0.8071 - val_accuracy: 0.7498
Epoch 64/100
210/210 [==============================] - 2s 9ms/step - loss: 0.7924 - accuracy: 0.7538 - val_loss: 0.8070 - val_accuracy: 0.7500
Epoch 65/100
210/210 [==============================] - 2s 9ms/step - loss: 0.7923 - accuracy: 0.7538 - val_loss: 0.8069 - val_accuracy: 0.7499
Epoch 66/100
210/210 [==============================] - 2s 9ms/step - loss: 0.7922 - accuracy: 0.7540 - val_loss: 0.8068 - val_accuracy: 0.7497
Epoch 67/100
210/210 [==============================] - 2s 11ms/step - loss: 0.7921 - accuracy: 0.7538 - val_loss: 0.8067 - val_accuracy: 0.7499
Epoch 68/100
210/210 [==============================] - 2s 11ms/step - loss: 0.7920 - accuracy: 0.7538 - val_loss: 0.8065 - val_accuracy: 0.7502
Epoch 69/100
210/210 [==============================] - 2s 9ms/step - loss: 0.7919 - accuracy: 0.7537 - val_loss: 0.8065 - val_accuracy: 0.7500
Epoch 70/100
210/210 [==============================] - 2s 9ms/step - loss: 0.7917 - accuracy: 0.7540 - val_loss: 0.8064 - val_accuracy: 0.7500
Epoch 71/100
210/210 [==============================] - 2s 9ms/step - loss: 0.7916 - accuracy: 0.7542 - val_loss: 0.8063 - val_accuracy: 0.7497
Epoch 72/100
210/210 [==============================] - 2s 9ms/step - loss: 0.7915 - accuracy: 0.7539 - val_loss: 0.8061 - val_accuracy: 0.7500
Epoch 73/100
210/210 [==============================] - 2s 11ms/step - loss: 0.7914 - accuracy: 0.7540 - val_loss: 0.8061 - val_accuracy: 0.7499
Epoch 74/100
210/210 [==============================] - 2s 9ms/step - loss: 0.7913 - accuracy: 0.7541 - val_loss: 0.8061 - val_accuracy: 0.7499
Epoch 75/100
210/210 [==============================] - 2s 9ms/step - loss: 0.7912 - accuracy: 0.7539 - val_loss: 0.8059 - val_accuracy: 0.7504
Epoch 76/100
210/210 [==============================] - 2s 9ms/step - loss: 0.7911 - accuracy: 0.7540 - val_loss: 0.8058 - val_accuracy: 0.7503
Epoch 77/100
210/210 [==============================] - 2s 9ms/step - loss: 0.7910 - accuracy: 0.7542 - val_loss: 0.8057 - val_accuracy: 0.7503
Epoch 78/100
210/210 [==============================] - 2s 11ms/step - loss: 0.7909 - accuracy: 0.7540 - val_loss: 0.8056 - val_accuracy: 0.7505
Epoch 79/100
210/210 [==============================] - 2s 11ms/step - loss: 0.7908 - accuracy: 0.7541 - val_loss: 0.8055 - val_accuracy: 0.7504
Epoch 80/100
210/210 [==============================] - 2s 9ms/step - loss: 0.7907 - accuracy: 0.7543 - val_loss: 0.8054 - val_accuracy: 0.7506
Epoch 81/100
210/210 [==============================] - 2s 9ms/step - loss: 0.7906 - accuracy: 0.7540 - val_loss: 0.8053 - val_accuracy: 0.7504
Epoch 82/100
210/210 [==============================] - 2s 11ms/step - loss: 0.7905 - accuracy: 0.7547 - val_loss: 0.8052 - val_accuracy: 0.7502
Epoch 83/100
210/210 [==============================] - 2s 9ms/step - loss: 0.7904 - accuracy: 0.7541 - val_loss: 0.8051 - val_accuracy: 0.7504
Epoch 84/100
210/210 [==============================] - 2s 9ms/step - loss: 0.7903 - accuracy: 0.7542 - val_loss: 0.8050 - val_accuracy: 0.7505
Epoch 85/100
210/210 [==============================] - 2s 11ms/step - loss: 0.7902 - accuracy: 0.7544 - val_loss: 0.8049 - val_accuracy: 0.7506
Epoch 86/100
210/210 [==============================] - 2s 9ms/step - loss: 0.7901 - accuracy: 0.7544 - val_loss: 0.8048 - val_accuracy: 0.7501
Epoch 87/100
210/210 [==============================] - 2s 11ms/step - loss: 0.7900 - accuracy: 0.7542 - val_loss: 0.8047 - val_accuracy: 0.7507
Epoch 88/100
210/210 [==============================] - 2s 11ms/step - loss: 0.7899 - accuracy: 0.7542 - val_loss: 0.8047 - val_accuracy: 0.7507
Epoch 89/100
210/210 [==============================] - 2s 11ms/step - loss: 0.7898 - accuracy: 0.7542 - val_loss: 0.8046 - val_accuracy: 0.7505
Epoch 90/100
210/210 [==============================] - 2s 9ms/step - loss: 0.7897 - accuracy: 0.7545 - val_loss: 0.8045 - val_accuracy: 0.7505
Epoch 91/100
210/210 [==============================] - 2s 11ms/step - loss: 0.7896 - accuracy: 0.7545 - val_loss: 0.8044 - val_accuracy: 0.7506
Epoch 92/100
210/210 [==============================] - 2s 11ms/step - loss: 0.7895 - accuracy: 0.7544 - val_loss: 0.8044 - val_accuracy: 0.7503
Epoch 93/100
210/210 [==============================] - 2s 9ms/step - loss: 0.7894 - accuracy: 0.7545 - val_loss: 0.8043 - val_accuracy: 0.7506
Epoch 94/100
210/210 [==============================] - 2s 9ms/step - loss: 0.7894 - accuracy: 0.7544 - val_loss: 0.8042 - val_accuracy: 0.7506
Epoch 95/100
210/210 [==============================] - 2s 11ms/step - loss: 0.7892 - accuracy: 0.7550 - val_loss: 0.8042 - val_accuracy: 0.7503
Epoch 96/100
210/210 [==============================] - 2s 11ms/step - loss: 0.7891 - accuracy: 0.7545 - val_loss: 0.8040 - val_accuracy: 0.7506
Epoch 97/100
210/210 [==============================] - 2s 11ms/step - loss: 0.7891 - accuracy: 0.7546 - val_loss: 0.8039 - val_accuracy: 0.7507
Epoch 98/100
210/210 [==============================] - 2s 9ms/step - loss: 0.7890 - accuracy: 0.7543 - val_loss: 0.8039 - val_accuracy: 0.7508
Epoch 99/100
210/210 [==============================] - 2s 11ms/step - loss: 0.7889 - accuracy: 0.7546 - val_loss: 0.8038 - val_accuracy: 0.7506
Epoch 100/100
210/210 [==============================] - 2s 9ms/step - loss: 0.7888 - accuracy: 0.7544 - val_loss: 0.8037 - val_accuracy: 0.7508
  • The accuracy of the model on val data using sgd with learning rate=0.0001 is improving a bit but not much.

Now, let's try to change the learning rate in ADAM optimizer

As, we are taking learning rate small means model will learn slowly. So, we have to take more number of epochs.

In [93]:
#Setting the learning rate  and decay in sgd
adam = optimizers.Adam(learning_rate = 0.0001)
# compiling the ANN classifier
keras_model.compile(optimizer = adam, loss = 'categorical_crossentropy', metrics = ['accuracy'])
# Fitting the ANN to the Training data
history = keras_model.fit(X_train, y_train, validation_data=(X_val,y_val),batch_size = 200, epochs = 100, verbose = 1)
Epoch 1/100
210/210 [==============================] - 3s 13ms/step - loss: 0.7982 - accuracy: 0.7522 - val_loss: 0.7999 - val_accuracy: 0.7522
Epoch 2/100
210/210 [==============================] - 2s 11ms/step - loss: 0.7891 - accuracy: 0.7558 - val_loss: 0.7993 - val_accuracy: 0.7523
Epoch 3/100
210/210 [==============================] - 2s 11ms/step - loss: 0.7846 - accuracy: 0.7563 - val_loss: 0.7931 - val_accuracy: 0.7551
Epoch 4/100
210/210 [==============================] - 2s 11ms/step - loss: 0.7797 - accuracy: 0.7587 - val_loss: 0.7890 - val_accuracy: 0.7558
Epoch 5/100
210/210 [==============================] - 2s 11ms/step - loss: 0.7728 - accuracy: 0.7597 - val_loss: 0.7830 - val_accuracy: 0.7585
Epoch 6/100
210/210 [==============================] - 2s 10ms/step - loss: 0.7678 - accuracy: 0.7613 - val_loss: 0.7775 - val_accuracy: 0.7600
Epoch 7/100
210/210 [==============================] - 2s 12ms/step - loss: 0.7628 - accuracy: 0.7641 - val_loss: 0.7810 - val_accuracy: 0.7583
Epoch 8/100
210/210 [==============================] - 2s 12ms/step - loss: 0.7583 - accuracy: 0.7642 - val_loss: 0.7793 - val_accuracy: 0.7593
Epoch 9/100
210/210 [==============================] - 2s 9ms/step - loss: 0.7524 - accuracy: 0.7669 - val_loss: 0.7655 - val_accuracy: 0.7648
Epoch 10/100
210/210 [==============================] - 4s 17ms/step - loss: 0.7489 - accuracy: 0.7674 - val_loss: 0.7605 - val_accuracy: 0.7652
Epoch 11/100
210/210 [==============================] - 2s 9ms/step - loss: 0.7426 - accuracy: 0.7701 - val_loss: 0.7580 - val_accuracy: 0.7667
Epoch 12/100
210/210 [==============================] - 2s 11ms/step - loss: 0.7385 - accuracy: 0.7709 - val_loss: 0.7513 - val_accuracy: 0.7689
Epoch 13/100
210/210 [==============================] - 2s 11ms/step - loss: 0.7344 - accuracy: 0.7728 - val_loss: 0.7485 - val_accuracy: 0.7691
Epoch 14/100
210/210 [==============================] - 2s 10ms/step - loss: 0.7294 - accuracy: 0.7745 - val_loss: 0.7459 - val_accuracy: 0.7701
Epoch 15/100
210/210 [==============================] - 2s 12ms/step - loss: 0.7270 - accuracy: 0.7744 - val_loss: 0.7428 - val_accuracy: 0.7713
Epoch 16/100
210/210 [==============================] - 2s 9ms/step - loss: 0.7214 - accuracy: 0.7759 - val_loss: 0.7384 - val_accuracy: 0.7725
Epoch 17/100
210/210 [==============================] - 2s 11ms/step - loss: 0.7186 - accuracy: 0.7774 - val_loss: 0.7335 - val_accuracy: 0.7748
Epoch 18/100
210/210 [==============================] - 2s 12ms/step - loss: 0.7151 - accuracy: 0.7786 - val_loss: 0.7305 - val_accuracy: 0.7758
Epoch 19/100
210/210 [==============================] - 2s 12ms/step - loss: 0.7104 - accuracy: 0.7800 - val_loss: 0.7337 - val_accuracy: 0.7739
Epoch 20/100
210/210 [==============================] - 2s 12ms/step - loss: 0.7076 - accuracy: 0.7809 - val_loss: 0.7212 - val_accuracy: 0.7782
Epoch 21/100
210/210 [==============================] - 2s 11ms/step - loss: 0.7026 - accuracy: 0.7828 - val_loss: 0.7269 - val_accuracy: 0.7760
Epoch 22/100
210/210 [==============================] - 2s 12ms/step - loss: 0.6990 - accuracy: 0.7829 - val_loss: 0.7155 - val_accuracy: 0.7803
Epoch 23/100
210/210 [==============================] - 2s 12ms/step - loss: 0.6959 - accuracy: 0.7832 - val_loss: 0.7095 - val_accuracy: 0.7824
Epoch 24/100
210/210 [==============================] - 2s 10ms/step - loss: 0.6926 - accuracy: 0.7865 - val_loss: 0.7115 - val_accuracy: 0.7807
Epoch 25/100
210/210 [==============================] - 2s 12ms/step - loss: 0.6884 - accuracy: 0.7871 - val_loss: 0.7166 - val_accuracy: 0.7792
Epoch 26/100
210/210 [==============================] - 2s 11ms/step - loss: 0.6858 - accuracy: 0.7870 - val_loss: 0.7052 - val_accuracy: 0.7831
Epoch 27/100
210/210 [==============================] - 2s 9ms/step - loss: 0.6832 - accuracy: 0.7870 - val_loss: 0.6988 - val_accuracy: 0.7847
Epoch 28/100
210/210 [==============================] - 2s 11ms/step - loss: 0.6793 - accuracy: 0.7889 - val_loss: 0.7017 - val_accuracy: 0.7852
Epoch 29/100
210/210 [==============================] - 2s 11ms/step - loss: 0.6743 - accuracy: 0.7906 - val_loss: 0.6951 - val_accuracy: 0.7870
Epoch 30/100
210/210 [==============================] - 2s 9ms/step - loss: 0.6720 - accuracy: 0.7921 - val_loss: 0.6913 - val_accuracy: 0.7876
Epoch 31/100
210/210 [==============================] - 2s 11ms/step - loss: 0.6686 - accuracy: 0.7925 - val_loss: 0.6942 - val_accuracy: 0.7863
Epoch 32/100
210/210 [==============================] - 2s 9ms/step - loss: 0.6653 - accuracy: 0.7941 - val_loss: 0.6917 - val_accuracy: 0.7862
Epoch 33/100
210/210 [==============================] - 2s 11ms/step - loss: 0.6642 - accuracy: 0.7941 - val_loss: 0.6799 - val_accuracy: 0.7914
Epoch 34/100
210/210 [==============================] - 2s 11ms/step - loss: 0.6609 - accuracy: 0.7949 - val_loss: 0.6832 - val_accuracy: 0.7908
Epoch 35/100
210/210 [==============================] - 2s 11ms/step - loss: 0.6567 - accuracy: 0.7960 - val_loss: 0.6833 - val_accuracy: 0.7898
Epoch 36/100
210/210 [==============================] - 2s 9ms/step - loss: 0.6540 - accuracy: 0.7975 - val_loss: 0.6781 - val_accuracy: 0.7925
Epoch 37/100
210/210 [==============================] - 2s 9ms/step - loss: 0.6501 - accuracy: 0.7986 - val_loss: 0.6781 - val_accuracy: 0.7921
Epoch 38/100
210/210 [==============================] - 2s 11ms/step - loss: 0.6470 - accuracy: 0.7999 - val_loss: 0.6669 - val_accuracy: 0.7957
Epoch 39/100
210/210 [==============================] - 2s 11ms/step - loss: 0.6456 - accuracy: 0.7997 - val_loss: 0.6754 - val_accuracy: 0.7913
Epoch 40/100
210/210 [==============================] - 2s 9ms/step - loss: 0.6421 - accuracy: 0.8006 - val_loss: 0.6700 - val_accuracy: 0.7944
Epoch 41/100
210/210 [==============================] - 2s 9ms/step - loss: 0.6390 - accuracy: 0.8016 - val_loss: 0.6619 - val_accuracy: 0.7976
Epoch 42/100
210/210 [==============================] - 2s 9ms/step - loss: 0.6352 - accuracy: 0.8037 - val_loss: 0.6557 - val_accuracy: 0.7994
Epoch 43/100
210/210 [==============================] - 2s 11ms/step - loss: 0.6312 - accuracy: 0.8052 - val_loss: 0.6578 - val_accuracy: 0.8000
Epoch 44/100
210/210 [==============================] - 2s 9ms/step - loss: 0.6288 - accuracy: 0.8059 - val_loss: 0.6559 - val_accuracy: 0.7997
Epoch 45/100
210/210 [==============================] - 2s 9ms/step - loss: 0.6261 - accuracy: 0.8065 - val_loss: 0.6540 - val_accuracy: 0.8001
Epoch 46/100
210/210 [==============================] - 2s 9ms/step - loss: 0.6230 - accuracy: 0.8080 - val_loss: 0.6504 - val_accuracy: 0.8006
Epoch 47/100
210/210 [==============================] - 2s 9ms/step - loss: 0.6212 - accuracy: 0.8068 - val_loss: 0.6516 - val_accuracy: 0.8003
Epoch 48/100
210/210 [==============================] - 2s 11ms/step - loss: 0.6195 - accuracy: 0.8081 - val_loss: 0.6471 - val_accuracy: 0.8033
Epoch 49/100
210/210 [==============================] - 2s 11ms/step - loss: 0.6169 - accuracy: 0.8082 - val_loss: 0.6465 - val_accuracy: 0.8023
Epoch 50/100
210/210 [==============================] - 2s 11ms/step - loss: 0.6136 - accuracy: 0.8086 - val_loss: 0.6421 - val_accuracy: 0.8044
Epoch 51/100
210/210 [==============================] - 2s 11ms/step - loss: 0.6112 - accuracy: 0.8103 - val_loss: 0.6397 - val_accuracy: 0.8052
Epoch 52/100
210/210 [==============================] - 2s 11ms/step - loss: 0.6093 - accuracy: 0.8125 - val_loss: 0.6331 - val_accuracy: 0.8065
Epoch 53/100
210/210 [==============================] - 2s 11ms/step - loss: 0.6069 - accuracy: 0.8123 - val_loss: 0.6383 - val_accuracy: 0.8051
Epoch 54/100
210/210 [==============================] - 2s 11ms/step - loss: 0.6021 - accuracy: 0.8136 - val_loss: 0.6333 - val_accuracy: 0.8069
Epoch 55/100
210/210 [==============================] - 2s 11ms/step - loss: 0.6023 - accuracy: 0.8131 - val_loss: 0.6303 - val_accuracy: 0.8077
Epoch 56/100
210/210 [==============================] - 2s 9ms/step - loss: 0.5980 - accuracy: 0.8144 - val_loss: 0.6315 - val_accuracy: 0.8077
Epoch 57/100
210/210 [==============================] - 2s 9ms/step - loss: 0.5947 - accuracy: 0.8159 - val_loss: 0.6373 - val_accuracy: 0.8041
Epoch 58/100
210/210 [==============================] - 2s 9ms/step - loss: 0.5935 - accuracy: 0.8163 - val_loss: 0.6212 - val_accuracy: 0.8112
Epoch 59/100
210/210 [==============================] - 2s 9ms/step - loss: 0.5908 - accuracy: 0.8162 - val_loss: 0.6233 - val_accuracy: 0.8102
Epoch 60/100
210/210 [==============================] - 2s 9ms/step - loss: 0.5877 - accuracy: 0.8187 - val_loss: 0.6215 - val_accuracy: 0.8107
Epoch 61/100
210/210 [==============================] - 2s 9ms/step - loss: 0.5853 - accuracy: 0.8185 - val_loss: 0.6168 - val_accuracy: 0.8124
Epoch 62/100
210/210 [==============================] - 2s 11ms/step - loss: 0.5847 - accuracy: 0.8176 - val_loss: 0.6262 - val_accuracy: 0.8077
Epoch 63/100
210/210 [==============================] - 2s 11ms/step - loss: 0.5801 - accuracy: 0.8199 - val_loss: 0.6215 - val_accuracy: 0.8101
Epoch 64/100
210/210 [==============================] - 2s 9ms/step - loss: 0.5801 - accuracy: 0.8190 - val_loss: 0.6150 - val_accuracy: 0.8119
Epoch 65/100
210/210 [==============================] - 2s 11ms/step - loss: 0.5760 - accuracy: 0.8218 - val_loss: 0.6083 - val_accuracy: 0.8145
Epoch 66/100
210/210 [==============================] - 2s 11ms/step - loss: 0.5746 - accuracy: 0.8213 - val_loss: 0.6094 - val_accuracy: 0.8147
Epoch 67/100
210/210 [==============================] - 2s 9ms/step - loss: 0.5731 - accuracy: 0.8225 - val_loss: 0.6023 - val_accuracy: 0.8169
Epoch 68/100
210/210 [==============================] - 2s 11ms/step - loss: 0.5706 - accuracy: 0.8229 - val_loss: 0.6048 - val_accuracy: 0.8152
Epoch 69/100
210/210 [==============================] - 2s 11ms/step - loss: 0.5667 - accuracy: 0.8239 - val_loss: 0.6006 - val_accuracy: 0.8177
Epoch 70/100
210/210 [==============================] - 2s 11ms/step - loss: 0.5669 - accuracy: 0.8247 - val_loss: 0.6032 - val_accuracy: 0.8156
Epoch 71/100
210/210 [==============================] - 2s 9ms/step - loss: 0.5612 - accuracy: 0.8268 - val_loss: 0.6068 - val_accuracy: 0.8136
Epoch 72/100
210/210 [==============================] - 2s 9ms/step - loss: 0.5611 - accuracy: 0.8257 - val_loss: 0.5938 - val_accuracy: 0.8198
Epoch 73/100
210/210 [==============================] - 2s 11ms/step - loss: 0.5582 - accuracy: 0.8272 - val_loss: 0.5908 - val_accuracy: 0.8205
Epoch 74/100
210/210 [==============================] - 2s 11ms/step - loss: 0.5565 - accuracy: 0.8272 - val_loss: 0.5937 - val_accuracy: 0.8196
Epoch 75/100
210/210 [==============================] - 2s 11ms/step - loss: 0.5543 - accuracy: 0.8268 - val_loss: 0.5921 - val_accuracy: 0.8191
Epoch 76/100
210/210 [==============================] - 2s 12ms/step - loss: 0.5527 - accuracy: 0.8285 - val_loss: 0.5928 - val_accuracy: 0.8197
Epoch 77/100
210/210 [==============================] - 2s 11ms/step - loss: 0.5494 - accuracy: 0.8302 - val_loss: 0.5918 - val_accuracy: 0.8193
Epoch 78/100
210/210 [==============================] - 2s 12ms/step - loss: 0.5479 - accuracy: 0.8294 - val_loss: 0.5901 - val_accuracy: 0.8205
Epoch 79/100
210/210 [==============================] - 2s 10ms/step - loss: 0.5457 - accuracy: 0.8308 - val_loss: 0.5889 - val_accuracy: 0.8202
Epoch 80/100
210/210 [==============================] - 2s 12ms/step - loss: 0.5441 - accuracy: 0.8323 - val_loss: 0.5884 - val_accuracy: 0.8207
Epoch 81/100
210/210 [==============================] - 2s 10ms/step - loss: 0.5412 - accuracy: 0.8325 - val_loss: 0.5868 - val_accuracy: 0.8205
Epoch 82/100
210/210 [==============================] - 2s 12ms/step - loss: 0.5414 - accuracy: 0.8317 - val_loss: 0.5828 - val_accuracy: 0.8219
Epoch 83/100
210/210 [==============================] - 2s 10ms/step - loss: 0.5375 - accuracy: 0.8343 - val_loss: 0.5847 - val_accuracy: 0.8217
Epoch 84/100
210/210 [==============================] - 2s 10ms/step - loss: 0.5350 - accuracy: 0.8345 - val_loss: 0.5771 - val_accuracy: 0.8239
Epoch 85/100
210/210 [==============================] - 2s 10ms/step - loss: 0.5339 - accuracy: 0.8341 - val_loss: 0.5763 - val_accuracy: 0.8251
Epoch 86/100
210/210 [==============================] - 2s 12ms/step - loss: 0.5303 - accuracy: 0.8356 - val_loss: 0.5805 - val_accuracy: 0.8233
Epoch 87/100
210/210 [==============================] - 2s 10ms/step - loss: 0.5286 - accuracy: 0.8368 - val_loss: 0.5692 - val_accuracy: 0.8277
Epoch 88/100
210/210 [==============================] - 2s 12ms/step - loss: 0.5259 - accuracy: 0.8369 - val_loss: 0.5729 - val_accuracy: 0.8260
Epoch 89/100
210/210 [==============================] - 2s 10ms/step - loss: 0.5261 - accuracy: 0.8366 - val_loss: 0.5728 - val_accuracy: 0.8263
Epoch 90/100
210/210 [==============================] - 2s 10ms/step - loss: 0.5228 - accuracy: 0.8378 - val_loss: 0.5660 - val_accuracy: 0.8277
Epoch 91/100
210/210 [==============================] - 2s 12ms/step - loss: 0.5203 - accuracy: 0.8387 - val_loss: 0.5623 - val_accuracy: 0.8298
Epoch 92/100
210/210 [==============================] - 2s 10ms/step - loss: 0.5188 - accuracy: 0.8400 - val_loss: 0.5681 - val_accuracy: 0.8265
Epoch 93/100
210/210 [==============================] - 2s 10ms/step - loss: 0.5173 - accuracy: 0.8400 - val_loss: 0.5598 - val_accuracy: 0.8308
Epoch 94/100
210/210 [==============================] - 2s 10ms/step - loss: 0.5149 - accuracy: 0.8399 - val_loss: 0.5651 - val_accuracy: 0.8287
Epoch 95/100
210/210 [==============================] - 2s 10ms/step - loss: 0.5147 - accuracy: 0.8398 - val_loss: 0.5622 - val_accuracy: 0.8295
Epoch 96/100
210/210 [==============================] - 2s 10ms/step - loss: 0.5123 - accuracy: 0.8415 - val_loss: 0.5643 - val_accuracy: 0.8279
Epoch 97/100
210/210 [==============================] - 2s 12ms/step - loss: 0.5097 - accuracy: 0.8410 - val_loss: 0.5583 - val_accuracy: 0.8316
Epoch 98/100
210/210 [==============================] - 2s 10ms/step - loss: 0.5079 - accuracy: 0.8429 - val_loss: 0.5573 - val_accuracy: 0.8305
Epoch 99/100
210/210 [==============================] - 2s 10ms/step - loss: 0.5057 - accuracy: 0.8433 - val_loss: 0.5542 - val_accuracy: 0.8319
Epoch 100/100
210/210 [==============================] - 2s 10ms/step - loss: 0.5038 - accuracy: 0.8426 - val_loss: 0.5569 - val_accuracy: 0.8313
  • The accuracy of the model on val data using ADAM with learning rate=0.0001 has shown significant improvement.
In [94]:
results = keras_model.evaluate(X_val, y_val)
print('Val_acc using simple NN adam : ', results[1])
1875/1875 [==============================] - 7s 3ms/step - loss: 0.5569 - accuracy: 0.8313
Val_acc using simple NN adam :  0.8313000202178955
In [95]:
#Store the accuracy results for each model in a dataframe for final comparison
results_on_val = pd.DataFrame({'Method':['NN'], 'accuracy': results[1]},index={'1'})
results_on_val = results_on_val[['Method', 'accuracy']]
results_on_val
Out[95]:
Method accuracy
1 NN 0.8313

Implementing batch normalization for training the neural network

In [96]:
##Initialize the Artificial Neural Network Classifier
keras_model_1 = Sequential()

Adding Batch normalization layer

In [97]:
# Input Layer
#Adding Input layer and activation functions ReLU
keras_model_1.add(Dense(512, kernel_initializer='he_normal',input_shape = (1024, )))
#Adding BatchNormalization Layer
keras_model_1.add(BatchNormalization())
#Adding Activation function
keras_model_1.add(Activation('relu'))

#Hidden Layer 1
#Adding first Hidden layer
keras_model_1.add(Dense(256, kernel_initializer='he_normal'))
#Adding BatchNormalization Layer
keras_model_1.add(BatchNormalization())
#Adding Activation function
keras_model_1.add(Activation('relu'))

#Hidden Layer 2
#Adding second Hidden layer
keras_model_1.add(Dense(128, kernel_initializer='he_normal'))
#Adding BatchNormalization Layer
keras_model_1.add(BatchNormalization())
#Adding Activation function
keras_model_1.add(Activation('relu'))

#Hidden Layer 3
#Adding third Hidden layer
keras_model_1.add(Dense(64, kernel_initializer='he_normal'))
#Adding BatchNormalization Layer
keras_model_1.add(BatchNormalization())
#Adding Activation function
keras_model_1.add(Activation('relu'))

#Hidden Layer 4
#Adding fourth Hidden layer
keras_model_1.add(Dense(32, kernel_initializer='he_normal'))
#Adding BatchNormalization Layer
keras_model_1.add(BatchNormalization())
#Adding Activation function
keras_model_1.add(Activation('relu'))

# Output Layer
#Adding output layer which is of 10 nodes (digidts)
keras_model_1.add(Dense(10))
#Adding Activation function
# Here, we are using softmax function because we have multiclass classsification
keras_model_1.add(Activation('softmax'))
In [98]:
keras_model_1.summary()
Model: "sequential_2"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 dense_10 (Dense)            (None, 512)               524800    
                                                                 
 batch_normalization (BatchN  (None, 512)              2048      
 ormalization)                                                   
                                                                 
 activation (Activation)     (None, 512)               0         
                                                                 
 dense_11 (Dense)            (None, 256)               131328    
                                                                 
 batch_normalization_1 (Batc  (None, 256)              1024      
 hNormalization)                                                 
                                                                 
 activation_1 (Activation)   (None, 256)               0         
                                                                 
 dense_12 (Dense)            (None, 128)               32896     
                                                                 
 batch_normalization_2 (Batc  (None, 128)              512       
 hNormalization)                                                 
                                                                 
 activation_2 (Activation)   (None, 128)               0         
                                                                 
 dense_13 (Dense)            (None, 64)                8256      
                                                                 
 batch_normalization_3 (Batc  (None, 64)               256       
 hNormalization)                                                 
                                                                 
 activation_3 (Activation)   (None, 64)                0         
                                                                 
 dense_14 (Dense)            (None, 32)                2080      
                                                                 
 batch_normalization_4 (Batc  (None, 32)               128       
 hNormalization)                                                 
                                                                 
 activation_4 (Activation)   (None, 32)                0         
                                                                 
 dense_15 (Dense)            (None, 10)                330       
                                                                 
 activation_5 (Activation)   (None, 10)                0         
                                                                 
=================================================================
Total params: 703,658
Trainable params: 701,674
Non-trainable params: 1,984
_________________________________________________________________

Using ADAM optimizer.

  • learning rate= 0.0001
In [99]:
#Setting the learning rate  and decay in sgd
adam = optimizers.Adam(learning_rate = 0.0001)
# compiling the ANN classifier
keras_model_1.compile(optimizer = adam, loss = 'categorical_crossentropy', metrics = ['accuracy'])
In [100]:
# Fitting the ANN to the Training data
history_1 = keras_model_1.fit(X_train, y_train, validation_data=(X_val,y_val),batch_size = 200, epochs = 100, verbose = 1)
Epoch 1/100
210/210 [==============================] - 5s 16ms/step - loss: 2.0952 - accuracy: 0.2660 - val_loss: 2.0480 - val_accuracy: 0.2981
Epoch 2/100
210/210 [==============================] - 3s 14ms/step - loss: 1.5532 - accuracy: 0.5215 - val_loss: 1.5619 - val_accuracy: 0.5263
Epoch 3/100
210/210 [==============================] - 3s 14ms/step - loss: 1.2684 - accuracy: 0.6414 - val_loss: 1.3287 - val_accuracy: 0.6105
Epoch 4/100
210/210 [==============================] - 3s 14ms/step - loss: 1.0817 - accuracy: 0.6971 - val_loss: 1.2046 - val_accuracy: 0.6461
Epoch 5/100
210/210 [==============================] - 3s 12ms/step - loss: 0.9479 - accuracy: 0.7335 - val_loss: 1.0199 - val_accuracy: 0.7077
Epoch 6/100
210/210 [==============================] - 3s 14ms/step - loss: 0.8464 - accuracy: 0.7605 - val_loss: 1.0016 - val_accuracy: 0.7132
Epoch 7/100
210/210 [==============================] - 2s 12ms/step - loss: 0.7647 - accuracy: 0.7818 - val_loss: 0.9833 - val_accuracy: 0.7055
Epoch 8/100
210/210 [==============================] - 2s 12ms/step - loss: 0.6996 - accuracy: 0.7987 - val_loss: 0.9263 - val_accuracy: 0.7149
Epoch 9/100
210/210 [==============================] - 3s 14ms/step - loss: 0.6463 - accuracy: 0.8131 - val_loss: 0.8426 - val_accuracy: 0.7515
Epoch 10/100
210/210 [==============================] - 3s 14ms/step - loss: 0.5987 - accuracy: 0.8252 - val_loss: 0.8753 - val_accuracy: 0.7323
Epoch 11/100
210/210 [==============================] - 3s 12ms/step - loss: 0.5590 - accuracy: 0.8377 - val_loss: 0.8677 - val_accuracy: 0.7331
Epoch 12/100
210/210 [==============================] - 3s 13ms/step - loss: 0.5231 - accuracy: 0.8469 - val_loss: 0.7806 - val_accuracy: 0.7592
Epoch 13/100
210/210 [==============================] - 3s 12ms/step - loss: 0.4922 - accuracy: 0.8550 - val_loss: 0.7760 - val_accuracy: 0.7609
Epoch 14/100
210/210 [==============================] - 3s 14ms/step - loss: 0.4595 - accuracy: 0.8648 - val_loss: 0.7525 - val_accuracy: 0.7637
Epoch 15/100
210/210 [==============================] - 3s 12ms/step - loss: 0.4315 - accuracy: 0.8742 - val_loss: 0.7343 - val_accuracy: 0.7731
Epoch 16/100
210/210 [==============================] - 3s 14ms/step - loss: 0.4124 - accuracy: 0.8781 - val_loss: 0.7666 - val_accuracy: 0.7656
Epoch 17/100
210/210 [==============================] - 3s 14ms/step - loss: 0.3888 - accuracy: 0.8862 - val_loss: 0.7502 - val_accuracy: 0.7750
Epoch 18/100
210/210 [==============================] - 3s 12ms/step - loss: 0.3684 - accuracy: 0.8930 - val_loss: 0.7458 - val_accuracy: 0.7676
Epoch 19/100
210/210 [==============================] - 3s 12ms/step - loss: 0.3493 - accuracy: 0.8979 - val_loss: 0.8191 - val_accuracy: 0.7497
Epoch 20/100
210/210 [==============================] - 3s 14ms/step - loss: 0.3297 - accuracy: 0.9041 - val_loss: 0.6114 - val_accuracy: 0.8113
Epoch 21/100
210/210 [==============================] - 2s 12ms/step - loss: 0.3146 - accuracy: 0.9086 - val_loss: 0.7260 - val_accuracy: 0.7747
Epoch 22/100
210/210 [==============================] - 3s 12ms/step - loss: 0.3001 - accuracy: 0.9130 - val_loss: 0.5758 - val_accuracy: 0.8205
Epoch 23/100
210/210 [==============================] - 3s 14ms/step - loss: 0.2826 - accuracy: 0.9173 - val_loss: 0.7048 - val_accuracy: 0.7780
Epoch 24/100
210/210 [==============================] - 3s 14ms/step - loss: 0.2721 - accuracy: 0.9209 - val_loss: 0.6790 - val_accuracy: 0.7925
Epoch 25/100
210/210 [==============================] - 3s 12ms/step - loss: 0.2560 - accuracy: 0.9259 - val_loss: 0.5821 - val_accuracy: 0.8205
Epoch 26/100
210/210 [==============================] - 3s 12ms/step - loss: 0.2468 - accuracy: 0.9280 - val_loss: 0.7079 - val_accuracy: 0.7817
Epoch 27/100
210/210 [==============================] - 3s 12ms/step - loss: 0.2352 - accuracy: 0.9312 - val_loss: 0.8238 - val_accuracy: 0.7612
Epoch 28/100
210/210 [==============================] - 3s 14ms/step - loss: 0.2244 - accuracy: 0.9357 - val_loss: 0.6718 - val_accuracy: 0.7938
Epoch 29/100
210/210 [==============================] - 3s 14ms/step - loss: 0.2148 - accuracy: 0.9385 - val_loss: 0.5988 - val_accuracy: 0.8184
Epoch 30/100
210/210 [==============================] - 3s 14ms/step - loss: 0.2094 - accuracy: 0.9397 - val_loss: 0.7127 - val_accuracy: 0.7868
Epoch 31/100
210/210 [==============================] - 3s 14ms/step - loss: 0.1961 - accuracy: 0.9441 - val_loss: 0.6898 - val_accuracy: 0.7882
Epoch 32/100
210/210 [==============================] - 3s 12ms/step - loss: 0.1921 - accuracy: 0.9440 - val_loss: 0.6465 - val_accuracy: 0.8075
Epoch 33/100
210/210 [==============================] - 3s 14ms/step - loss: 0.1806 - accuracy: 0.9481 - val_loss: 0.6137 - val_accuracy: 0.8168
Epoch 34/100
210/210 [==============================] - 3s 14ms/step - loss: 0.1758 - accuracy: 0.9494 - val_loss: 0.6055 - val_accuracy: 0.8167
Epoch 35/100
210/210 [==============================] - 3s 14ms/step - loss: 0.1690 - accuracy: 0.9508 - val_loss: 0.7783 - val_accuracy: 0.7731
Epoch 36/100
210/210 [==============================] - 2s 12ms/step - loss: 0.1633 - accuracy: 0.9528 - val_loss: 0.6952 - val_accuracy: 0.8044
Epoch 37/100
210/210 [==============================] - 3s 14ms/step - loss: 0.1529 - accuracy: 0.9568 - val_loss: 0.6625 - val_accuracy: 0.8145
Epoch 38/100
210/210 [==============================] - 3s 12ms/step - loss: 0.1551 - accuracy: 0.9552 - val_loss: 0.6335 - val_accuracy: 0.8187
Epoch 39/100
210/210 [==============================] - 3s 14ms/step - loss: 0.1404 - accuracy: 0.9598 - val_loss: 0.7159 - val_accuracy: 0.7941
Epoch 40/100
210/210 [==============================] - 3s 13ms/step - loss: 0.1384 - accuracy: 0.9603 - val_loss: 0.5939 - val_accuracy: 0.8265
Epoch 41/100
210/210 [==============================] - 3s 12ms/step - loss: 0.1342 - accuracy: 0.9616 - val_loss: 0.6739 - val_accuracy: 0.8095
Epoch 42/100
210/210 [==============================] - 3s 14ms/step - loss: 0.1254 - accuracy: 0.9646 - val_loss: 0.7371 - val_accuracy: 0.8011
Epoch 43/100
210/210 [==============================] - 3s 14ms/step - loss: 0.1214 - accuracy: 0.9656 - val_loss: 0.5984 - val_accuracy: 0.8309
Epoch 44/100
210/210 [==============================] - 3s 14ms/step - loss: 0.1297 - accuracy: 0.9625 - val_loss: 0.6709 - val_accuracy: 0.8083
Epoch 45/100
210/210 [==============================] - 3s 14ms/step - loss: 0.1158 - accuracy: 0.9670 - val_loss: 0.6505 - val_accuracy: 0.8184
Epoch 46/100
210/210 [==============================] - 3s 12ms/step - loss: 0.1133 - accuracy: 0.9674 - val_loss: 0.7658 - val_accuracy: 0.7892
Epoch 47/100
210/210 [==============================] - 3s 14ms/step - loss: 0.1078 - accuracy: 0.9690 - val_loss: 0.6844 - val_accuracy: 0.8035
Epoch 48/100
210/210 [==============================] - 3s 14ms/step - loss: 0.1048 - accuracy: 0.9703 - val_loss: 0.6415 - val_accuracy: 0.8224
Epoch 49/100
210/210 [==============================] - 3s 14ms/step - loss: 0.1040 - accuracy: 0.9698 - val_loss: 0.7493 - val_accuracy: 0.7988
Epoch 50/100
210/210 [==============================] - 3s 12ms/step - loss: 0.1034 - accuracy: 0.9695 - val_loss: 0.5581 - val_accuracy: 0.8408
Epoch 51/100
210/210 [==============================] - 3s 12ms/step - loss: 0.1034 - accuracy: 0.9696 - val_loss: 0.7060 - val_accuracy: 0.8132
Epoch 52/100
210/210 [==============================] - 3s 14ms/step - loss: 0.0945 - accuracy: 0.9725 - val_loss: 0.8202 - val_accuracy: 0.7878
Epoch 53/100
210/210 [==============================] - 3s 14ms/step - loss: 0.0948 - accuracy: 0.9721 - val_loss: 0.6062 - val_accuracy: 0.8258
Epoch 54/100
210/210 [==============================] - 3s 14ms/step - loss: 0.0917 - accuracy: 0.9731 - val_loss: 0.5070 - val_accuracy: 0.8583
Epoch 55/100
210/210 [==============================] - 3s 12ms/step - loss: 0.0906 - accuracy: 0.9738 - val_loss: 0.5946 - val_accuracy: 0.8375
Epoch 56/100
210/210 [==============================] - 2s 12ms/step - loss: 0.0859 - accuracy: 0.9752 - val_loss: 0.8064 - val_accuracy: 0.8056
Epoch 57/100
210/210 [==============================] - 3s 14ms/step - loss: 0.0871 - accuracy: 0.9742 - val_loss: 0.5610 - val_accuracy: 0.8456
Epoch 58/100
210/210 [==============================] - 3s 14ms/step - loss: 0.0815 - accuracy: 0.9759 - val_loss: 0.6554 - val_accuracy: 0.8278
Epoch 59/100
210/210 [==============================] - 3s 14ms/step - loss: 0.0903 - accuracy: 0.9730 - val_loss: 0.7110 - val_accuracy: 0.8155
Epoch 60/100
210/210 [==============================] - 3s 12ms/step - loss: 0.0829 - accuracy: 0.9749 - val_loss: 0.7155 - val_accuracy: 0.8210
Epoch 61/100
210/210 [==============================] - 3s 12ms/step - loss: 0.0756 - accuracy: 0.9781 - val_loss: 0.7876 - val_accuracy: 0.8037
Epoch 62/100
210/210 [==============================] - 3s 14ms/step - loss: 0.0778 - accuracy: 0.9778 - val_loss: 0.7339 - val_accuracy: 0.8099
Epoch 63/100
210/210 [==============================] - 3s 12ms/step - loss: 0.0698 - accuracy: 0.9796 - val_loss: 0.5544 - val_accuracy: 0.8513
Epoch 64/100
210/210 [==============================] - 3s 14ms/step - loss: 0.0676 - accuracy: 0.9809 - val_loss: 0.5792 - val_accuracy: 0.8451
Epoch 65/100
210/210 [==============================] - 3s 12ms/step - loss: 0.0620 - accuracy: 0.9822 - val_loss: 0.6082 - val_accuracy: 0.8458
Epoch 66/100
210/210 [==============================] - 3s 14ms/step - loss: 0.0680 - accuracy: 0.9798 - val_loss: 0.4974 - val_accuracy: 0.8679
Epoch 67/100
210/210 [==============================] - 3s 12ms/step - loss: 0.0696 - accuracy: 0.9786 - val_loss: 0.5870 - val_accuracy: 0.8435
Epoch 68/100
210/210 [==============================] - 3s 14ms/step - loss: 0.0745 - accuracy: 0.9780 - val_loss: 0.7624 - val_accuracy: 0.8123
Epoch 69/100
210/210 [==============================] - 3s 14ms/step - loss: 0.0632 - accuracy: 0.9817 - val_loss: 0.7142 - val_accuracy: 0.8190
Epoch 70/100
210/210 [==============================] - 3s 12ms/step - loss: 0.0668 - accuracy: 0.9798 - val_loss: 0.5855 - val_accuracy: 0.8494
Epoch 71/100
210/210 [==============================] - 3s 12ms/step - loss: 0.0595 - accuracy: 0.9833 - val_loss: 0.8540 - val_accuracy: 0.7876
Epoch 72/100
210/210 [==============================] - 3s 14ms/step - loss: 0.0675 - accuracy: 0.9795 - val_loss: 0.8477 - val_accuracy: 0.8097
Epoch 73/100
210/210 [==============================] - 3s 14ms/step - loss: 0.0657 - accuracy: 0.9805 - val_loss: 0.7155 - val_accuracy: 0.8306
Epoch 74/100
210/210 [==============================] - 3s 14ms/step - loss: 0.0626 - accuracy: 0.9816 - val_loss: 0.6740 - val_accuracy: 0.8351
Epoch 75/100
210/210 [==============================] - 3s 14ms/step - loss: 0.0500 - accuracy: 0.9862 - val_loss: 0.5456 - val_accuracy: 0.8559
Epoch 76/100
210/210 [==============================] - 2s 12ms/step - loss: 0.0617 - accuracy: 0.9818 - val_loss: 0.6664 - val_accuracy: 0.8349
Epoch 77/100
210/210 [==============================] - 3s 12ms/step - loss: 0.0612 - accuracy: 0.9817 - val_loss: 0.9089 - val_accuracy: 0.7947
Epoch 78/100
210/210 [==============================] - 3s 14ms/step - loss: 0.0625 - accuracy: 0.9824 - val_loss: 0.9595 - val_accuracy: 0.8017
Epoch 79/100
210/210 [==============================] - 3s 14ms/step - loss: 0.0552 - accuracy: 0.9831 - val_loss: 0.7440 - val_accuracy: 0.8209
Epoch 80/100
210/210 [==============================] - 3s 14ms/step - loss: 0.0598 - accuracy: 0.9817 - val_loss: 0.6496 - val_accuracy: 0.8408
Epoch 81/100
210/210 [==============================] - 3s 12ms/step - loss: 0.0555 - accuracy: 0.9837 - val_loss: 0.5547 - val_accuracy: 0.8573
Epoch 82/100
210/210 [==============================] - 3s 14ms/step - loss: 0.0560 - accuracy: 0.9836 - val_loss: 0.7075 - val_accuracy: 0.8283
Epoch 83/100
210/210 [==============================] - 3s 12ms/step - loss: 0.0489 - accuracy: 0.9859 - val_loss: 0.6299 - val_accuracy: 0.8515
Epoch 84/100
210/210 [==============================] - 3s 14ms/step - loss: 0.0499 - accuracy: 0.9852 - val_loss: 1.1039 - val_accuracy: 0.7752
Epoch 85/100
210/210 [==============================] - 3s 12ms/step - loss: 0.0510 - accuracy: 0.9841 - val_loss: 0.7479 - val_accuracy: 0.8254
Epoch 86/100
210/210 [==============================] - 3s 12ms/step - loss: 0.0632 - accuracy: 0.9810 - val_loss: 0.6448 - val_accuracy: 0.8438
Epoch 87/100
210/210 [==============================] - 3s 12ms/step - loss: 0.0589 - accuracy: 0.9823 - val_loss: 0.6341 - val_accuracy: 0.8524
Epoch 88/100
210/210 [==============================] - 3s 14ms/step - loss: 0.0519 - accuracy: 0.9848 - val_loss: 0.6474 - val_accuracy: 0.8416
Epoch 89/100
210/210 [==============================] - 3s 12ms/step - loss: 0.0502 - accuracy: 0.9853 - val_loss: 0.6825 - val_accuracy: 0.8369
Epoch 90/100
210/210 [==============================] - 3s 12ms/step - loss: 0.0462 - accuracy: 0.9861 - val_loss: 0.5900 - val_accuracy: 0.8576
Epoch 91/100
210/210 [==============================] - 3s 14ms/step - loss: 0.0478 - accuracy: 0.9858 - val_loss: 0.6940 - val_accuracy: 0.8378
Epoch 92/100
210/210 [==============================] - 3s 12ms/step - loss: 0.0446 - accuracy: 0.9877 - val_loss: 0.8255 - val_accuracy: 0.8240
Epoch 93/100
210/210 [==============================] - 3s 12ms/step - loss: 0.0391 - accuracy: 0.9887 - val_loss: 0.9899 - val_accuracy: 0.7879
Epoch 94/100
210/210 [==============================] - 3s 12ms/step - loss: 0.0532 - accuracy: 0.9842 - val_loss: 0.6991 - val_accuracy: 0.8362
Epoch 95/100
210/210 [==============================] - 3s 14ms/step - loss: 0.0459 - accuracy: 0.9864 - val_loss: 0.7548 - val_accuracy: 0.8247
Epoch 96/100
210/210 [==============================] - 3s 14ms/step - loss: 0.0522 - accuracy: 0.9838 - val_loss: 1.1284 - val_accuracy: 0.7610
Epoch 97/100
210/210 [==============================] - 3s 12ms/step - loss: 0.0471 - accuracy: 0.9862 - val_loss: 0.6754 - val_accuracy: 0.8447
Epoch 98/100
210/210 [==============================] - 3s 14ms/step - loss: 0.0438 - accuracy: 0.9874 - val_loss: 0.6349 - val_accuracy: 0.8522
Epoch 99/100
210/210 [==============================] - 3s 14ms/step - loss: 0.0420 - accuracy: 0.9870 - val_loss: 0.6474 - val_accuracy: 0.8470
Epoch 100/100
210/210 [==============================] - 3s 14ms/step - loss: 0.0364 - accuracy: 0.9890 - val_loss: 0.8959 - val_accuracy: 0.8187
In [101]:
results_1 = keras_model_1.evaluate(X_val, y_val)
print('Val_acc using Batch ANN adam : ', results_1[1])
1875/1875 [==============================] - 8s 4ms/step - loss: 0.8959 - accuracy: 0.8187
Val_acc using Batch ANN adam :  0.8186666369438171
In [102]:
#Store the accuracy results for each model in a dataframe for final comparison
tempResultsDf = pd.DataFrame({'Method':['NN_Batch_adam'], 'accuracy': [results_1[1]]},index={'2'})
results_on_val= pd.concat([results_on_val, tempResultsDf])
results_on_val = results_on_val[['Method', 'accuracy']]
results_on_val
Out[102]:
Method accuracy
1 NN 0.831300
2 NN_Batch_adam 0.818667

Implementing (vectorized) backpropagation (cost stochastic gradient descent, cross entropy loss, cost functions)

In [103]:
#Initialize the Artificial Neural Network Classifier
keras_model_2 = Sequential()

Adding Batch normalization layer

In [104]:
# Input Layer
#Adding Input layer and activation functions ReLU
keras_model_2.add(Dense(512, kernel_initializer='he_normal',input_shape = (1024, )))
#Adding BatchNormalization Layer
keras_model_2.add(BatchNormalization())
#Adding Activation function
keras_model_2.add(Activation('relu'))

#Hidden Layer 1
#Adding first Hidden layer
keras_model_2.add(Dense(256, kernel_initializer='he_normal'))
#Adding BatchNormalization Layer
keras_model_2.add(BatchNormalization())
#Adding Activation function
keras_model_2.add(Activation('relu'))

#Hidden Layer 2
#Adding second Hidden layer
keras_model_2.add(Dense(128, kernel_initializer='he_normal'))
#Adding BatchNormalization Layer
keras_model_2.add(BatchNormalization())
#Adding Activation function
keras_model_2.add(Activation('relu'))

#Hidden Layer 3
#Adding third Hidden layer
keras_model_2.add(Dense(64, kernel_initializer='he_normal'))
#Adding BatchNormalization Layer
keras_model_2.add(BatchNormalization())
#Adding Activation function
keras_model_2.add(Activation('relu'))

#Hidden Layer 4
#Adding fourth Hidden layer
keras_model_2.add(Dense(32, kernel_initializer='he_normal'))
#Adding BatchNormalization Layer
keras_model_2.add(BatchNormalization())
#Adding Activation function
keras_model_2.add(Activation('relu'))

# Output Layer
#Adding output layer which is of 10 nodes (digidts)
keras_model_2.add(Dense(10))
#Adding Activation function
# Here, we are using softmax function because we have multiclass classsification
keras_model_2.add(Activation('softmax'))
In [105]:
keras_model_2.summary()
Model: "sequential_3"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 dense_16 (Dense)            (None, 512)               524800    
                                                                 
 batch_normalization_5 (Batc  (None, 512)              2048      
 hNormalization)                                                 
                                                                 
 activation_6 (Activation)   (None, 512)               0         
                                                                 
 dense_17 (Dense)            (None, 256)               131328    
                                                                 
 batch_normalization_6 (Batc  (None, 256)              1024      
 hNormalization)                                                 
                                                                 
 activation_7 (Activation)   (None, 256)               0         
                                                                 
 dense_18 (Dense)            (None, 128)               32896     
                                                                 
 batch_normalization_7 (Batc  (None, 128)              512       
 hNormalization)                                                 
                                                                 
 activation_8 (Activation)   (None, 128)               0         
                                                                 
 dense_19 (Dense)            (None, 64)                8256      
                                                                 
 batch_normalization_8 (Batc  (None, 64)               256       
 hNormalization)                                                 
                                                                 
 activation_9 (Activation)   (None, 64)                0         
                                                                 
 dense_20 (Dense)            (None, 32)                2080      
                                                                 
 batch_normalization_9 (Batc  (None, 32)               128       
 hNormalization)                                                 
                                                                 
 activation_10 (Activation)  (None, 32)                0         
                                                                 
 dense_21 (Dense)            (None, 10)                330       
                                                                 
 activation_11 (Activation)  (None, 10)                0         
                                                                 
=================================================================
Total params: 703,658
Trainable params: 701,674
Non-trainable params: 1,984
_________________________________________________________________

Using SGD optimizer

  • learning rate =0.0001
In [106]:
#Setting the learning rate in sgd
sgd = optimizers.SGD(learning_rate = 0.0001)
# compiling the ANN classifier
keras_model_2.compile(optimizer = sgd, loss = 'categorical_crossentropy', metrics = ['accuracy'])
In [107]:
# Fitting the ANN to the Training data
history_2 = keras_model_2.fit(X_train, y_train, validation_data=(X_val,y_val),batch_size = 200, epochs = 10, verbose = 1)
Epoch 1/10
210/210 [==============================] - 4s 15ms/step - loss: 2.5409 - accuracy: 0.1035 - val_loss: 2.4033 - val_accuracy: 0.1034
Epoch 2/10
210/210 [==============================] - 3s 13ms/step - loss: 2.5271 - accuracy: 0.1066 - val_loss: 2.4795 - val_accuracy: 0.1073
Epoch 3/10
210/210 [==============================] - 2s 11ms/step - loss: 2.5130 - accuracy: 0.1074 - val_loss: 2.5025 - val_accuracy: 0.1099
Epoch 4/10
210/210 [==============================] - 2s 11ms/step - loss: 2.4985 - accuracy: 0.1103 - val_loss: 2.4938 - val_accuracy: 0.1125
Epoch 5/10
210/210 [==============================] - 3s 13ms/step - loss: 2.4849 - accuracy: 0.1142 - val_loss: 2.4819 - val_accuracy: 0.1150
Epoch 6/10
210/210 [==============================] - 3s 13ms/step - loss: 2.4741 - accuracy: 0.1156 - val_loss: 2.4708 - val_accuracy: 0.1179
Epoch 7/10
210/210 [==============================] - 2s 11ms/step - loss: 2.4605 - accuracy: 0.1184 - val_loss: 2.4589 - val_accuracy: 0.1202
Epoch 8/10
210/210 [==============================] - 2s 11ms/step - loss: 2.4492 - accuracy: 0.1218 - val_loss: 2.4467 - val_accuracy: 0.1233
Epoch 9/10
210/210 [==============================] - 3s 13ms/step - loss: 2.4370 - accuracy: 0.1256 - val_loss: 2.4358 - val_accuracy: 0.1264
Epoch 10/10
210/210 [==============================] - 3s 13ms/step - loss: 2.4261 - accuracy: 0.1262 - val_loss: 2.4251 - val_accuracy: 0.1293
In [109]:
results_2 = keras_model_2.evaluate(X_val, y_val)
print('Val_acc using Batch ANN adam : ', results_2[1])
1875/1875 [==============================] - 12s 6ms/step - loss: 2.4251 - accuracy: 0.1293
Val_acc using Batch ANN adam :  0.1292833387851715
In [110]:
#Store the accuracy results for each model in a dataframe for final comparison
tempResultsDf = pd.DataFrame({'Method':['NN_Batch_sgd'], 'accuracy': [results_2[1]]},index={'3'})
results_on_val= pd.concat([results_on_val, tempResultsDf])
results_on_val = results_on_val[['Method', 'accuracy']]
results_on_val
Out[110]:
Method accuracy
1 NN 0.831300
2 NN_Batch_adam 0.818667
3 NN_Batch_sgd 0.129283
  • The batch normalization using SGD optimizer is giving better results than using adam optimizer.

Implementing batch normalization and Dropout for training the neural network

In [111]:
##Initialize the Artificial Neural Network Classifier
keras_model_3 = Sequential()

Adding BatchNormalization Layer and Dropout Layer (0.5)

In [112]:
# Input Layer
#Adding Input layer and activation functions ReLU
keras_model_3.add(Dense(512, kernel_initializer='he_normal',input_shape = (1024, )))
#Adding BatchNormalization Layer
keras_model_3.add(BatchNormalization())
#Adding Activation function
keras_model_3.add(Activation('relu'))
#Adding Dropout Layer
keras_model_3.add(Dropout(0.5))

#Hidden Layer 1
#Adding first Hidden layer
keras_model_3.add(Dense(256,kernel_initializer='he_normal'))
#Adding BatchNormalization Layer
keras_model_3.add(BatchNormalization())
#Adding Activation function
keras_model_3.add(Activation('relu'))
#Adding Dropout Layer
keras_model_3.add(Dropout(0.5))

#Hidden Layer 2
#Adding second Hidden layer
keras_model_3.add(Dense(128, kernel_initializer='he_normal'))
#Adding BatchNormalization Layer
keras_model_3.add(BatchNormalization())
#Adding Activation function
keras_model_3.add(Activation('relu'))
#Adding Dropout Layer
keras_model_3.add(Dropout(0.5))

#Hidden Layer 3
#Adding third Hidden layer
keras_model_3.add(Dense(64, kernel_initializer='he_normal'))
#Adding BatchNormalization Layer
keras_model_3.add(BatchNormalization())
#Adding Activation function
keras_model_3.add(Activation('relu'))
#Adding Dropout Layer
keras_model_3.add(Dropout(0.5))

#Hidden Layer 4
#Adding fourth Hidden layer
keras_model_3.add(Dense(32, kernel_initializer='he_normal'))
#Adding BatchNormalization Layer
keras_model_3.add(BatchNormalization())
#Adding Activation function
keras_model_3.add(Activation('relu'))
#Adding Dropout Layer
keras_model_3.add(Dropout(0.5))

# Output Layer
#Adding output layer which is of 10 nodes (digidts)
keras_model_3.add(Dense(10))
#Adding Activation function
# Here, we are using softmax function because we have multiclass classsification
keras_model_3.add(Activation('softmax'))
In [113]:
keras_model_3.summary()
Model: "sequential_4"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 dense_22 (Dense)            (None, 512)               524800    
                                                                 
 batch_normalization_10 (Bat  (None, 512)              2048      
 chNormalization)                                                
                                                                 
 activation_12 (Activation)  (None, 512)               0         
                                                                 
 dropout (Dropout)           (None, 512)               0         
                                                                 
 dense_23 (Dense)            (None, 256)               131328    
                                                                 
 batch_normalization_11 (Bat  (None, 256)              1024      
 chNormalization)                                                
                                                                 
 activation_13 (Activation)  (None, 256)               0         
                                                                 
 dropout_1 (Dropout)         (None, 256)               0         
                                                                 
 dense_24 (Dense)            (None, 128)               32896     
                                                                 
 batch_normalization_12 (Bat  (None, 128)              512       
 chNormalization)                                                
                                                                 
 activation_14 (Activation)  (None, 128)               0         
                                                                 
 dropout_2 (Dropout)         (None, 128)               0         
                                                                 
 dense_25 (Dense)            (None, 64)                8256      
                                                                 
 batch_normalization_13 (Bat  (None, 64)               256       
 chNormalization)                                                
                                                                 
 activation_15 (Activation)  (None, 64)                0         
                                                                 
 dropout_3 (Dropout)         (None, 64)                0         
                                                                 
 dense_26 (Dense)            (None, 32)                2080      
                                                                 
 batch_normalization_14 (Bat  (None, 32)               128       
 chNormalization)                                                
                                                                 
 activation_16 (Activation)  (None, 32)                0         
                                                                 
 dropout_4 (Dropout)         (None, 32)                0         
                                                                 
 dense_27 (Dense)            (None, 10)                330       
                                                                 
 activation_17 (Activation)  (None, 10)                0         
                                                                 
=================================================================
Total params: 703,658
Trainable params: 701,674
Non-trainable params: 1,984
_________________________________________________________________

Using ADAM optimizer.

  • learning rate= 0.0001
In [114]:
#Setting the learning rate in adam
adam = optimizers.Adam(learning_rate = 0.0001)
# compiling the ANN classifier
keras_model_3.compile(optimizer = adam, loss = 'categorical_crossentropy', metrics = ['accuracy'])
In [115]:
# Fitting the ANN to the Training data
history_3 = keras_model_3.fit(X_train, y_train, validation_data=(X_val,y_val),batch_size = 200, epochs = 100, verbose = 1)
Epoch 1/100
210/210 [==============================] - 5s 15ms/step - loss: 2.8546 - accuracy: 0.0982 - val_loss: 2.3275 - val_accuracy: 0.1027
Epoch 2/100
210/210 [==============================] - 3s 13ms/step - loss: 2.7133 - accuracy: 0.1021 - val_loss: 2.3296 - val_accuracy: 0.1049
Epoch 3/100
210/210 [==============================] - 3s 14ms/step - loss: 2.6078 - accuracy: 0.1042 - val_loss: 2.3233 - val_accuracy: 0.1068
Epoch 4/100
210/210 [==============================] - 3s 12ms/step - loss: 2.5423 - accuracy: 0.1021 - val_loss: 2.3164 - val_accuracy: 0.1076
Epoch 5/100
210/210 [==============================] - 3s 14ms/step - loss: 2.4925 - accuracy: 0.1026 - val_loss: 2.3122 - val_accuracy: 0.1059
Epoch 6/100
210/210 [==============================] - 3s 14ms/step - loss: 2.4467 - accuracy: 0.1020 - val_loss: 2.3071 - val_accuracy: 0.1061
Epoch 7/100
210/210 [==============================] - 3s 13ms/step - loss: 2.4222 - accuracy: 0.1027 - val_loss: 2.3023 - val_accuracy: 0.1088
Epoch 8/100
210/210 [==============================] - 3s 13ms/step - loss: 2.3952 - accuracy: 0.1047 - val_loss: 2.3009 - val_accuracy: 0.1081
Epoch 9/100
210/210 [==============================] - 3s 13ms/step - loss: 2.3729 - accuracy: 0.1032 - val_loss: 2.2978 - val_accuracy: 0.1092
Epoch 10/100
210/210 [==============================] - 3s 12ms/step - loss: 2.3603 - accuracy: 0.1063 - val_loss: 2.2975 - val_accuracy: 0.1179
Epoch 11/100
210/210 [==============================] - 3s 14ms/step - loss: 2.3468 - accuracy: 0.1088 - val_loss: 2.2936 - val_accuracy: 0.1243
Epoch 12/100
210/210 [==============================] - 3s 12ms/step - loss: 2.3365 - accuracy: 0.1093 - val_loss: 2.2933 - val_accuracy: 0.1358
Epoch 13/100
210/210 [==============================] - 3s 14ms/step - loss: 2.3328 - accuracy: 0.1060 - val_loss: 2.2892 - val_accuracy: 0.1448
Epoch 14/100
210/210 [==============================] - 3s 14ms/step - loss: 2.3239 - accuracy: 0.1092 - val_loss: 2.2853 - val_accuracy: 0.1616
Epoch 15/100
210/210 [==============================] - 3s 14ms/step - loss: 2.3190 - accuracy: 0.1119 - val_loss: 2.2820 - val_accuracy: 0.1635
Epoch 16/100
210/210 [==============================] - 3s 12ms/step - loss: 2.3118 - accuracy: 0.1136 - val_loss: 2.2780 - val_accuracy: 0.1892
Epoch 17/100
210/210 [==============================] - 3s 14ms/step - loss: 2.3034 - accuracy: 0.1218 - val_loss: 2.2636 - val_accuracy: 0.2182
Epoch 18/100
210/210 [==============================] - 3s 14ms/step - loss: 2.2922 - accuracy: 0.1258 - val_loss: 2.2498 - val_accuracy: 0.2368
Epoch 19/100
210/210 [==============================] - 3s 12ms/step - loss: 2.2812 - accuracy: 0.1336 - val_loss: 2.2219 - val_accuracy: 0.2380
Epoch 20/100
210/210 [==============================] - 3s 14ms/step - loss: 2.2599 - accuracy: 0.1465 - val_loss: 2.1888 - val_accuracy: 0.2396
Epoch 21/100
210/210 [==============================] - 3s 14ms/step - loss: 2.2337 - accuracy: 0.1602 - val_loss: 2.1636 - val_accuracy: 0.2195
Epoch 22/100
210/210 [==============================] - 3s 14ms/step - loss: 2.2019 - accuracy: 0.1717 - val_loss: 2.1243 - val_accuracy: 0.2247
Epoch 23/100
210/210 [==============================] - 3s 12ms/step - loss: 2.1686 - accuracy: 0.1865 - val_loss: 2.0658 - val_accuracy: 0.2458
Epoch 24/100
210/210 [==============================] - 3s 14ms/step - loss: 2.1310 - accuracy: 0.2026 - val_loss: 2.0336 - val_accuracy: 0.2671
Epoch 25/100
210/210 [==============================] - 3s 12ms/step - loss: 2.1019 - accuracy: 0.2061 - val_loss: 1.9877 - val_accuracy: 0.2923
Epoch 26/100
210/210 [==============================] - 3s 12ms/step - loss: 2.0675 - accuracy: 0.2225 - val_loss: 1.9402 - val_accuracy: 0.3022
Epoch 27/100
210/210 [==============================] - 3s 14ms/step - loss: 2.0382 - accuracy: 0.2308 - val_loss: 1.9054 - val_accuracy: 0.3048
Epoch 28/100
210/210 [==============================] - 3s 14ms/step - loss: 2.0118 - accuracy: 0.2374 - val_loss: 1.9250 - val_accuracy: 0.2784
Epoch 29/100
210/210 [==============================] - 3s 12ms/step - loss: 1.9872 - accuracy: 0.2483 - val_loss: 1.8568 - val_accuracy: 0.3214
Epoch 30/100
210/210 [==============================] - 3s 14ms/step - loss: 1.9596 - accuracy: 0.2548 - val_loss: 1.8272 - val_accuracy: 0.3359
Epoch 31/100
210/210 [==============================] - 3s 12ms/step - loss: 1.9371 - accuracy: 0.2639 - val_loss: 1.8133 - val_accuracy: 0.3409
Epoch 32/100
210/210 [==============================] - 3s 14ms/step - loss: 1.9163 - accuracy: 0.2749 - val_loss: 1.7937 - val_accuracy: 0.3441
Epoch 33/100
210/210 [==============================] - 3s 13ms/step - loss: 1.8902 - accuracy: 0.2805 - val_loss: 1.7725 - val_accuracy: 0.3493
Epoch 34/100
210/210 [==============================] - 3s 12ms/step - loss: 1.8689 - accuracy: 0.2934 - val_loss: 1.7043 - val_accuracy: 0.3924
Epoch 35/100
210/210 [==============================] - 3s 14ms/step - loss: 1.8453 - accuracy: 0.3020 - val_loss: 1.6922 - val_accuracy: 0.4063
Epoch 36/100
210/210 [==============================] - 3s 13ms/step - loss: 1.8268 - accuracy: 0.3095 - val_loss: 1.6765 - val_accuracy: 0.4103
Epoch 37/100
210/210 [==============================] - 3s 14ms/step - loss: 1.8049 - accuracy: 0.3176 - val_loss: 1.6399 - val_accuracy: 0.4330
Epoch 38/100
210/210 [==============================] - 3s 12ms/step - loss: 1.7854 - accuracy: 0.3263 - val_loss: 1.6947 - val_accuracy: 0.3982
Epoch 39/100
210/210 [==============================] - 3s 13ms/step - loss: 1.7613 - accuracy: 0.3385 - val_loss: 1.6165 - val_accuracy: 0.4389
Epoch 40/100
210/210 [==============================] - 3s 12ms/step - loss: 1.7402 - accuracy: 0.3449 - val_loss: 1.5769 - val_accuracy: 0.4575
Epoch 41/100
210/210 [==============================] - 3s 14ms/step - loss: 1.7193 - accuracy: 0.3565 - val_loss: 1.4994 - val_accuracy: 0.4928
Epoch 42/100
210/210 [==============================] - 3s 14ms/step - loss: 1.6993 - accuracy: 0.3635 - val_loss: 1.5517 - val_accuracy: 0.4605
Epoch 43/100
210/210 [==============================] - 3s 12ms/step - loss: 1.6787 - accuracy: 0.3745 - val_loss: 1.4574 - val_accuracy: 0.5121
Epoch 44/100
210/210 [==============================] - 3s 12ms/step - loss: 1.6630 - accuracy: 0.3783 - val_loss: 1.4295 - val_accuracy: 0.5165
Epoch 45/100
210/210 [==============================] - 3s 14ms/step - loss: 1.6426 - accuracy: 0.3834 - val_loss: 1.4136 - val_accuracy: 0.5185
Epoch 46/100
210/210 [==============================] - 3s 12ms/step - loss: 1.6223 - accuracy: 0.3927 - val_loss: 1.3794 - val_accuracy: 0.5286
Epoch 47/100
210/210 [==============================] - 3s 14ms/step - loss: 1.6159 - accuracy: 0.3980 - val_loss: 1.4452 - val_accuracy: 0.4948
Epoch 48/100
210/210 [==============================] - 3s 14ms/step - loss: 1.5936 - accuracy: 0.4036 - val_loss: 1.3689 - val_accuracy: 0.5264
Epoch 49/100
210/210 [==============================] - 3s 12ms/step - loss: 1.5827 - accuracy: 0.4054 - val_loss: 1.4583 - val_accuracy: 0.4868
Epoch 50/100
210/210 [==============================] - 3s 12ms/step - loss: 1.5658 - accuracy: 0.4142 - val_loss: 1.4006 - val_accuracy: 0.5051
Epoch 51/100
210/210 [==============================] - 3s 14ms/step - loss: 1.5557 - accuracy: 0.4161 - val_loss: 1.3715 - val_accuracy: 0.5136
Epoch 52/100
210/210 [==============================] - 3s 12ms/step - loss: 1.5384 - accuracy: 0.4267 - val_loss: 1.3295 - val_accuracy: 0.5394
Epoch 53/100
210/210 [==============================] - 3s 12ms/step - loss: 1.5305 - accuracy: 0.4304 - val_loss: 1.2882 - val_accuracy: 0.5474
Epoch 54/100
210/210 [==============================] - 3s 12ms/step - loss: 1.5250 - accuracy: 0.4306 - val_loss: 1.3192 - val_accuracy: 0.5451
Epoch 55/100
210/210 [==============================] - 3s 14ms/step - loss: 1.5143 - accuracy: 0.4339 - val_loss: 1.3396 - val_accuracy: 0.5293
Epoch 56/100
210/210 [==============================] - 3s 12ms/step - loss: 1.5025 - accuracy: 0.4434 - val_loss: 1.2649 - val_accuracy: 0.5674
Epoch 57/100
210/210 [==============================] - 3s 14ms/step - loss: 1.4879 - accuracy: 0.4445 - val_loss: 1.2653 - val_accuracy: 0.5700
Epoch 58/100
210/210 [==============================] - 3s 14ms/step - loss: 1.4776 - accuracy: 0.4527 - val_loss: 1.2225 - val_accuracy: 0.5829
Epoch 59/100
210/210 [==============================] - 3s 12ms/step - loss: 1.4716 - accuracy: 0.4559 - val_loss: 1.2877 - val_accuracy: 0.5613
Epoch 60/100
210/210 [==============================] - 3s 14ms/step - loss: 1.4614 - accuracy: 0.4562 - val_loss: 1.2190 - val_accuracy: 0.5783
Epoch 61/100
210/210 [==============================] - 3s 14ms/step - loss: 1.4491 - accuracy: 0.4660 - val_loss: 1.3162 - val_accuracy: 0.5484
Epoch 62/100
210/210 [==============================] - 3s 14ms/step - loss: 1.4400 - accuracy: 0.4697 - val_loss: 1.2522 - val_accuracy: 0.5559
Epoch 63/100
210/210 [==============================] - 3s 12ms/step - loss: 1.4332 - accuracy: 0.4729 - val_loss: 1.1954 - val_accuracy: 0.5892
Epoch 64/100
210/210 [==============================] - 3s 14ms/step - loss: 1.4287 - accuracy: 0.4776 - val_loss: 1.2334 - val_accuracy: 0.5749
Epoch 65/100
210/210 [==============================] - 3s 12ms/step - loss: 1.4132 - accuracy: 0.4824 - val_loss: 1.1674 - val_accuracy: 0.6083
Epoch 66/100
210/210 [==============================] - 3s 12ms/step - loss: 1.4095 - accuracy: 0.4862 - val_loss: 1.1859 - val_accuracy: 0.5856
Epoch 67/100
210/210 [==============================] - 3s 12ms/step - loss: 1.3986 - accuracy: 0.4928 - val_loss: 1.1635 - val_accuracy: 0.6045
Epoch 68/100
210/210 [==============================] - 3s 12ms/step - loss: 1.3820 - accuracy: 0.4981 - val_loss: 1.1596 - val_accuracy: 0.6158
Epoch 69/100
210/210 [==============================] - 3s 12ms/step - loss: 1.3785 - accuracy: 0.4991 - val_loss: 1.1370 - val_accuracy: 0.6332
Epoch 70/100
210/210 [==============================] - 3s 12ms/step - loss: 1.3692 - accuracy: 0.4998 - val_loss: 1.2293 - val_accuracy: 0.5596
Epoch 71/100
210/210 [==============================] - 3s 12ms/step - loss: 1.3640 - accuracy: 0.5062 - val_loss: 1.1605 - val_accuracy: 0.5855
Epoch 72/100
210/210 [==============================] - 3s 12ms/step - loss: 1.3458 - accuracy: 0.5130 - val_loss: 1.0896 - val_accuracy: 0.6357
Epoch 73/100
210/210 [==============================] - 3s 14ms/step - loss: 1.3449 - accuracy: 0.5158 - val_loss: 1.0850 - val_accuracy: 0.6247
Epoch 74/100
210/210 [==============================] - 3s 14ms/step - loss: 1.3345 - accuracy: 0.5218 - val_loss: 1.1023 - val_accuracy: 0.6262
Epoch 75/100
210/210 [==============================] - 3s 12ms/step - loss: 1.3272 - accuracy: 0.5241 - val_loss: 1.1144 - val_accuracy: 0.6286
Epoch 76/100
210/210 [==============================] - 3s 12ms/step - loss: 1.3161 - accuracy: 0.5283 - val_loss: 1.3215 - val_accuracy: 0.5551
Epoch 77/100
210/210 [==============================] - 3s 14ms/step - loss: 1.3128 - accuracy: 0.5306 - val_loss: 1.0838 - val_accuracy: 0.6598
Epoch 78/100
210/210 [==============================] - 3s 14ms/step - loss: 1.3027 - accuracy: 0.5386 - val_loss: 1.1343 - val_accuracy: 0.6274
Epoch 79/100
210/210 [==============================] - 3s 12ms/step - loss: 1.2905 - accuracy: 0.5383 - val_loss: 1.0043 - val_accuracy: 0.6699
Epoch 80/100
210/210 [==============================] - 3s 12ms/step - loss: 1.2911 - accuracy: 0.5409 - val_loss: 0.9994 - val_accuracy: 0.6797
Epoch 81/100
210/210 [==============================] - 3s 14ms/step - loss: 1.2845 - accuracy: 0.5422 - val_loss: 1.0895 - val_accuracy: 0.6536
Epoch 82/100
210/210 [==============================] - 3s 12ms/step - loss: 1.2761 - accuracy: 0.5467 - val_loss: 1.0021 - val_accuracy: 0.6759
Epoch 83/100
210/210 [==============================] - 3s 12ms/step - loss: 1.2679 - accuracy: 0.5560 - val_loss: 1.0344 - val_accuracy: 0.6733
Epoch 84/100
210/210 [==============================] - 3s 14ms/step - loss: 1.2570 - accuracy: 0.5548 - val_loss: 1.0057 - val_accuracy: 0.6856
Epoch 85/100
210/210 [==============================] - 3s 12ms/step - loss: 1.2460 - accuracy: 0.5571 - val_loss: 1.0833 - val_accuracy: 0.6380
Epoch 86/100
210/210 [==============================] - 3s 12ms/step - loss: 1.2378 - accuracy: 0.5622 - val_loss: 1.0290 - val_accuracy: 0.6798
Epoch 87/100
210/210 [==============================] - 3s 14ms/step - loss: 1.2448 - accuracy: 0.5617 - val_loss: 0.9632 - val_accuracy: 0.6907
Epoch 88/100
210/210 [==============================] - 3s 12ms/step - loss: 1.2216 - accuracy: 0.5682 - val_loss: 1.2013 - val_accuracy: 0.5721
Epoch 89/100
210/210 [==============================] - 3s 12ms/step - loss: 1.2225 - accuracy: 0.5686 - val_loss: 0.9764 - val_accuracy: 0.6825
Epoch 90/100
210/210 [==============================] - 3s 12ms/step - loss: 1.2203 - accuracy: 0.5694 - val_loss: 1.0314 - val_accuracy: 0.6520
Epoch 91/100
210/210 [==============================] - 3s 12ms/step - loss: 1.2100 - accuracy: 0.5772 - val_loss: 0.9255 - val_accuracy: 0.7100
Epoch 92/100
210/210 [==============================] - 3s 13ms/step - loss: 1.2019 - accuracy: 0.5820 - val_loss: 0.9344 - val_accuracy: 0.7148
Epoch 93/100
210/210 [==============================] - 3s 12ms/step - loss: 1.1976 - accuracy: 0.5828 - val_loss: 0.9402 - val_accuracy: 0.6973
Epoch 94/100
210/210 [==============================] - 3s 14ms/step - loss: 1.1882 - accuracy: 0.5860 - val_loss: 0.9343 - val_accuracy: 0.6987
Epoch 95/100
210/210 [==============================] - 3s 12ms/step - loss: 1.1838 - accuracy: 0.5888 - val_loss: 0.9491 - val_accuracy: 0.7124
Epoch 96/100
210/210 [==============================] - 3s 14ms/step - loss: 1.1742 - accuracy: 0.5875 - val_loss: 0.9080 - val_accuracy: 0.7314
Epoch 97/100
210/210 [==============================] - 3s 12ms/step - loss: 1.1783 - accuracy: 0.5898 - val_loss: 0.9759 - val_accuracy: 0.7008
Epoch 98/100
210/210 [==============================] - 3s 14ms/step - loss: 1.1629 - accuracy: 0.5969 - val_loss: 0.9177 - val_accuracy: 0.7209
Epoch 99/100
210/210 [==============================] - 3s 14ms/step - loss: 1.1637 - accuracy: 0.5963 - val_loss: 0.9129 - val_accuracy: 0.7210
Epoch 100/100
210/210 [==============================] - 3s 12ms/step - loss: 1.1566 - accuracy: 0.5996 - val_loss: 0.9700 - val_accuracy: 0.7052
In [116]:
results_3 = keras_model_3.evaluate(X_val, y_val)
print('Val_acc using BatchNorm and Dropout adam : ', results_3[1])
1875/1875 [==============================] - 8s 4ms/step - loss: 0.9700 - accuracy: 0.7052
Val_acc using BatchNorm and Dropout adam :  0.7052333354949951
In [117]:
#Store the accuracy results for each model in a dataframe for final comparison
tempResultsDf = pd.DataFrame({'Method':['NN_Batch_Drop_adam'], 'accuracy': [results_3[1]]},index={'4'})
results_on_val= pd.concat([results_on_val, tempResultsDf])
results_on_val = results_on_val[['Method', 'accuracy']]
results_on_val
Out[117]:
Method accuracy
1 NN 0.831300
2 NN_Batch_adam 0.818667
3 NN_Batch_sgd 0.129283
4 NN_Batch_Drop_adam 0.705233

Testing NN on test data

In [118]:
##Initialize the Artificial Neural Network Classifier
keras_model_final = Sequential()
In [119]:
# Input Layer
#Adding Input layer and activation functions ReLU
keras_model_final.add(Dense(512, kernel_initializer='he_normal',input_shape = (1024, )))
#Adding BatchNormalization Layer
keras_model_final.add(BatchNormalization())
#Adding Activation function
keras_model_final.add(Activation('relu'))

#Hidden Layer 1
#Adding first Hidden layer
keras_model_final.add(Dense(256, kernel_initializer='he_normal'))
#Adding BatchNormalization Layer
keras_model_final.add(BatchNormalization())
#Adding Activation function
keras_model_final.add(Activation('relu'))

#Hidden Layer 2
#Adding second Hidden layer
keras_model_final.add(Dense(128, kernel_initializer='he_normal'))
#Adding BatchNormalization Layer
keras_model_final.add(BatchNormalization())
#Adding Activation function
keras_model_final.add(Activation('relu'))

#Hidden Layer 3
#Adding third Hidden layer
keras_model_final.add(Dense(64, kernel_initializer='he_normal'))
#Adding BatchNormalization Layer
keras_model_final.add(BatchNormalization())
#Adding Activation function
keras_model_final.add(Activation('relu'))

#Hidden Layer 4
#Adding fourth Hidden layer
keras_model_final.add(Dense(32, kernel_initializer='he_normal'))
#Adding BatchNormalization Layer
keras_model_final.add(BatchNormalization())
#Adding Activation function
keras_model_final.add(Activation('relu'))

# Output Layer
#Adding output layer which is of 10 nodes (digidts)
keras_model_final.add(Dense(10))
#Adding Activation function
# Here, we are using softmax function because we have multiclass classsification
keras_model_final.add(Activation('softmax'))
In [120]:
keras_model_final.summary()
Model: "sequential_5"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 dense_28 (Dense)            (None, 512)               524800    
                                                                 
 batch_normalization_15 (Bat  (None, 512)              2048      
 chNormalization)                                                
                                                                 
 activation_18 (Activation)  (None, 512)               0         
                                                                 
 dense_29 (Dense)            (None, 256)               131328    
                                                                 
 batch_normalization_16 (Bat  (None, 256)              1024      
 chNormalization)                                                
                                                                 
 activation_19 (Activation)  (None, 256)               0         
                                                                 
 dense_30 (Dense)            (None, 128)               32896     
                                                                 
 batch_normalization_17 (Bat  (None, 128)              512       
 chNormalization)                                                
                                                                 
 activation_20 (Activation)  (None, 128)               0         
                                                                 
 dense_31 (Dense)            (None, 64)                8256      
                                                                 
 batch_normalization_18 (Bat  (None, 64)               256       
 chNormalization)                                                
                                                                 
 activation_21 (Activation)  (None, 64)                0         
                                                                 
 dense_32 (Dense)            (None, 32)                2080      
                                                                 
 batch_normalization_19 (Bat  (None, 32)               128       
 chNormalization)                                                
                                                                 
 activation_22 (Activation)  (None, 32)                0         
                                                                 
 dense_33 (Dense)            (None, 10)                330       
                                                                 
 activation_23 (Activation)  (None, 10)                0         
                                                                 
=================================================================
Total params: 703,658
Trainable params: 701,674
Non-trainable params: 1,984
_________________________________________________________________
In [121]:
#Setting the learning rate in sgd
adam = optimizers.Adam(learning_rate = 0.0001)
# compiling the ANN classifier
keras_model_final.compile(optimizer = adam, loss = 'categorical_crossentropy', metrics = ['accuracy'])
In [122]:
# Fitting the ANN to the Training data
history_final = keras_model_final.fit(X_train, y_train, validation_data=(X_test,y_test),batch_size = 200, epochs = 100, verbose = 1)
Epoch 1/100
210/210 [==============================] - 4s 10ms/step - loss: 2.1181 - accuracy: 0.2396 - val_loss: 2.0814 - val_accuracy: 0.2542
Epoch 2/100
210/210 [==============================] - 2s 9ms/step - loss: 1.5900 - accuracy: 0.5101 - val_loss: 1.6359 - val_accuracy: 0.4955
Epoch 3/100
210/210 [==============================] - 2s 9ms/step - loss: 1.2936 - accuracy: 0.6366 - val_loss: 1.3779 - val_accuracy: 0.5762
Epoch 4/100
210/210 [==============================] - 2s 9ms/step - loss: 1.0982 - accuracy: 0.6981 - val_loss: 1.2484 - val_accuracy: 0.6315
Epoch 5/100
210/210 [==============================] - 2s 9ms/step - loss: 0.9616 - accuracy: 0.7314 - val_loss: 1.1837 - val_accuracy: 0.6542
Epoch 6/100
210/210 [==============================] - 2s 9ms/step - loss: 0.8565 - accuracy: 0.7583 - val_loss: 1.1261 - val_accuracy: 0.6650
Epoch 7/100
210/210 [==============================] - 2s 9ms/step - loss: 0.7730 - accuracy: 0.7798 - val_loss: 1.0619 - val_accuracy: 0.6657
Epoch 8/100
210/210 [==============================] - 2s 9ms/step - loss: 0.7051 - accuracy: 0.7977 - val_loss: 0.9848 - val_accuracy: 0.7002
Epoch 9/100
210/210 [==============================] - 2s 9ms/step - loss: 0.6507 - accuracy: 0.8112 - val_loss: 0.9716 - val_accuracy: 0.7011
Epoch 10/100
210/210 [==============================] - 2s 9ms/step - loss: 0.6031 - accuracy: 0.8246 - val_loss: 0.9538 - val_accuracy: 0.7022
Epoch 11/100
210/210 [==============================] - 2s 9ms/step - loss: 0.5566 - accuracy: 0.8381 - val_loss: 0.9662 - val_accuracy: 0.7002
Epoch 12/100
210/210 [==============================] - 2s 9ms/step - loss: 0.5218 - accuracy: 0.8483 - val_loss: 0.9660 - val_accuracy: 0.7003
Epoch 13/100
210/210 [==============================] - 2s 9ms/step - loss: 0.4909 - accuracy: 0.8561 - val_loss: 0.9177 - val_accuracy: 0.7144
Epoch 14/100
210/210 [==============================] - 2s 9ms/step - loss: 0.4588 - accuracy: 0.8651 - val_loss: 0.8739 - val_accuracy: 0.7285
Epoch 15/100
210/210 [==============================] - 2s 9ms/step - loss: 0.4328 - accuracy: 0.8736 - val_loss: 0.8489 - val_accuracy: 0.7401
Epoch 16/100
210/210 [==============================] - 2s 9ms/step - loss: 0.4059 - accuracy: 0.8816 - val_loss: 0.9044 - val_accuracy: 0.7175
Epoch 17/100
210/210 [==============================] - 2s 9ms/step - loss: 0.3835 - accuracy: 0.8880 - val_loss: 1.0105 - val_accuracy: 0.6936
Epoch 18/100
210/210 [==============================] - 2s 9ms/step - loss: 0.3608 - accuracy: 0.8958 - val_loss: 0.8794 - val_accuracy: 0.7313
Epoch 19/100
210/210 [==============================] - 2s 9ms/step - loss: 0.3427 - accuracy: 0.9006 - val_loss: 0.9527 - val_accuracy: 0.7113
Epoch 20/100
210/210 [==============================] - 2s 9ms/step - loss: 0.3283 - accuracy: 0.9040 - val_loss: 1.1223 - val_accuracy: 0.6806
Epoch 21/100
210/210 [==============================] - 2s 9ms/step - loss: 0.3103 - accuracy: 0.9095 - val_loss: 1.1278 - val_accuracy: 0.6772
Epoch 22/100
210/210 [==============================] - 2s 9ms/step - loss: 0.2928 - accuracy: 0.9151 - val_loss: 0.8368 - val_accuracy: 0.7472
Epoch 23/100
210/210 [==============================] - 2s 9ms/step - loss: 0.2819 - accuracy: 0.9199 - val_loss: 0.8496 - val_accuracy: 0.7500
Epoch 24/100
210/210 [==============================] - 2s 9ms/step - loss: 0.2658 - accuracy: 0.9231 - val_loss: 0.8885 - val_accuracy: 0.7370
Epoch 25/100
210/210 [==============================] - 2s 9ms/step - loss: 0.2526 - accuracy: 0.9275 - val_loss: 0.9275 - val_accuracy: 0.7387
Epoch 26/100
210/210 [==============================] - 2s 9ms/step - loss: 0.2426 - accuracy: 0.9309 - val_loss: 0.9715 - val_accuracy: 0.7307
Epoch 27/100
210/210 [==============================] - 2s 9ms/step - loss: 0.2272 - accuracy: 0.9352 - val_loss: 0.8518 - val_accuracy: 0.7472
Epoch 28/100
210/210 [==============================] - 2s 9ms/step - loss: 0.2210 - accuracy: 0.9371 - val_loss: 0.8683 - val_accuracy: 0.7508
Epoch 29/100
210/210 [==============================] - 2s 9ms/step - loss: 0.2108 - accuracy: 0.9396 - val_loss: 0.8698 - val_accuracy: 0.7439
Epoch 30/100
210/210 [==============================] - 2s 9ms/step - loss: 0.2010 - accuracy: 0.9430 - val_loss: 0.8775 - val_accuracy: 0.7490
Epoch 31/100
210/210 [==============================] - 2s 9ms/step - loss: 0.1934 - accuracy: 0.9455 - val_loss: 0.8936 - val_accuracy: 0.7502
Epoch 32/100
210/210 [==============================] - 2s 9ms/step - loss: 0.1810 - accuracy: 0.9484 - val_loss: 0.9793 - val_accuracy: 0.7356
Epoch 33/100
210/210 [==============================] - 2s 9ms/step - loss: 0.1768 - accuracy: 0.9500 - val_loss: 0.9172 - val_accuracy: 0.7416
Epoch 34/100
210/210 [==============================] - 2s 9ms/step - loss: 0.1711 - accuracy: 0.9504 - val_loss: 0.9762 - val_accuracy: 0.7404
Epoch 35/100
210/210 [==============================] - 2s 9ms/step - loss: 0.1640 - accuracy: 0.9536 - val_loss: 1.0934 - val_accuracy: 0.7191
Epoch 36/100
210/210 [==============================] - 2s 9ms/step - loss: 0.1594 - accuracy: 0.9549 - val_loss: 0.8975 - val_accuracy: 0.7549
Epoch 37/100
210/210 [==============================] - 2s 9ms/step - loss: 0.1504 - accuracy: 0.9574 - val_loss: 1.0811 - val_accuracy: 0.7183
Epoch 38/100
210/210 [==============================] - 2s 9ms/step - loss: 0.1484 - accuracy: 0.9579 - val_loss: 0.9902 - val_accuracy: 0.7362
Epoch 39/100
210/210 [==============================] - 2s 9ms/step - loss: 0.1406 - accuracy: 0.9602 - val_loss: 1.1544 - val_accuracy: 0.7211
Epoch 40/100
210/210 [==============================] - 2s 9ms/step - loss: 0.1369 - accuracy: 0.9614 - val_loss: 1.1761 - val_accuracy: 0.7033
Epoch 41/100
210/210 [==============================] - 2s 9ms/step - loss: 0.1320 - accuracy: 0.9624 - val_loss: 1.0173 - val_accuracy: 0.7322
Epoch 42/100
210/210 [==============================] - 2s 9ms/step - loss: 0.1255 - accuracy: 0.9652 - val_loss: 1.1132 - val_accuracy: 0.7261
Epoch 43/100
210/210 [==============================] - 2s 9ms/step - loss: 0.1196 - accuracy: 0.9665 - val_loss: 1.0447 - val_accuracy: 0.7446
Epoch 44/100
210/210 [==============================] - 2s 9ms/step - loss: 0.1189 - accuracy: 0.9650 - val_loss: 1.2747 - val_accuracy: 0.6983
Epoch 45/100
210/210 [==============================] - 2s 9ms/step - loss: 0.1159 - accuracy: 0.9670 - val_loss: 1.1605 - val_accuracy: 0.7117
Epoch 46/100
210/210 [==============================] - 2s 9ms/step - loss: 0.1087 - accuracy: 0.9702 - val_loss: 0.9378 - val_accuracy: 0.7587
Epoch 47/100
210/210 [==============================] - 2s 9ms/step - loss: 0.1083 - accuracy: 0.9693 - val_loss: 1.0492 - val_accuracy: 0.7381
Epoch 48/100
210/210 [==============================] - 2s 9ms/step - loss: 0.1070 - accuracy: 0.9693 - val_loss: 1.0212 - val_accuracy: 0.7538
Epoch 49/100
210/210 [==============================] - 2s 9ms/step - loss: 0.1043 - accuracy: 0.9695 - val_loss: 0.9837 - val_accuracy: 0.7496
Epoch 50/100
210/210 [==============================] - 2s 9ms/step - loss: 0.1017 - accuracy: 0.9704 - val_loss: 1.0435 - val_accuracy: 0.7496
Epoch 51/100
210/210 [==============================] - 2s 9ms/step - loss: 0.0998 - accuracy: 0.9713 - val_loss: 1.1111 - val_accuracy: 0.7360
Epoch 52/100
210/210 [==============================] - 2s 9ms/step - loss: 0.0938 - accuracy: 0.9732 - val_loss: 1.3019 - val_accuracy: 0.7011
Epoch 53/100
210/210 [==============================] - 2s 9ms/step - loss: 0.0936 - accuracy: 0.9733 - val_loss: 0.9672 - val_accuracy: 0.7686
Epoch 54/100
210/210 [==============================] - 2s 9ms/step - loss: 0.0872 - accuracy: 0.9756 - val_loss: 1.0846 - val_accuracy: 0.7441
Epoch 55/100
210/210 [==============================] - 2s 9ms/step - loss: 0.0859 - accuracy: 0.9765 - val_loss: 1.1172 - val_accuracy: 0.7363
Epoch 56/100
210/210 [==============================] - 2s 9ms/step - loss: 0.0880 - accuracy: 0.9738 - val_loss: 1.1228 - val_accuracy: 0.7421
Epoch 57/100
210/210 [==============================] - 2s 9ms/step - loss: 0.0780 - accuracy: 0.9775 - val_loss: 1.1424 - val_accuracy: 0.7337
Epoch 58/100
210/210 [==============================] - 2s 9ms/step - loss: 0.0836 - accuracy: 0.9758 - val_loss: 1.1006 - val_accuracy: 0.7349
Epoch 59/100
210/210 [==============================] - 2s 9ms/step - loss: 0.0777 - accuracy: 0.9772 - val_loss: 1.2056 - val_accuracy: 0.7172
Epoch 60/100
210/210 [==============================] - 2s 9ms/step - loss: 0.0761 - accuracy: 0.9782 - val_loss: 1.1891 - val_accuracy: 0.7344
Epoch 61/100
210/210 [==============================] - 2s 9ms/step - loss: 0.0815 - accuracy: 0.9767 - val_loss: 1.2306 - val_accuracy: 0.7244
Epoch 62/100
210/210 [==============================] - 2s 9ms/step - loss: 0.0775 - accuracy: 0.9776 - val_loss: 1.1826 - val_accuracy: 0.7429
Epoch 63/100
210/210 [==============================] - 2s 9ms/step - loss: 0.0757 - accuracy: 0.9776 - val_loss: 1.1764 - val_accuracy: 0.7344
Epoch 64/100
210/210 [==============================] - 2s 9ms/step - loss: 0.0706 - accuracy: 0.9797 - val_loss: 1.1291 - val_accuracy: 0.7489
Epoch 65/100
210/210 [==============================] - 2s 9ms/step - loss: 0.0676 - accuracy: 0.9804 - val_loss: 1.1414 - val_accuracy: 0.7508
Epoch 66/100
210/210 [==============================] - 2s 9ms/step - loss: 0.0704 - accuracy: 0.9800 - val_loss: 1.2159 - val_accuracy: 0.7365
Epoch 67/100
210/210 [==============================] - 2s 9ms/step - loss: 0.0697 - accuracy: 0.9791 - val_loss: 1.2460 - val_accuracy: 0.7194
Epoch 68/100
210/210 [==============================] - 2s 9ms/step - loss: 0.0737 - accuracy: 0.9777 - val_loss: 1.5636 - val_accuracy: 0.6956
Epoch 69/100
210/210 [==============================] - 2s 9ms/step - loss: 0.0689 - accuracy: 0.9794 - val_loss: 1.2826 - val_accuracy: 0.7296
Epoch 70/100
210/210 [==============================] - 2s 9ms/step - loss: 0.0671 - accuracy: 0.9805 - val_loss: 1.1836 - val_accuracy: 0.7437
Epoch 71/100
210/210 [==============================] - 2s 9ms/step - loss: 0.0589 - accuracy: 0.9836 - val_loss: 1.4259 - val_accuracy: 0.7048
Epoch 72/100
210/210 [==============================] - 2s 9ms/step - loss: 0.0598 - accuracy: 0.9825 - val_loss: 1.2176 - val_accuracy: 0.7248
Epoch 73/100
210/210 [==============================] - 2s 9ms/step - loss: 0.0637 - accuracy: 0.9812 - val_loss: 1.2615 - val_accuracy: 0.7273
Epoch 74/100
210/210 [==============================] - 2s 9ms/step - loss: 0.0646 - accuracy: 0.9808 - val_loss: 1.3545 - val_accuracy: 0.7298
Epoch 75/100
210/210 [==============================] - 2s 9ms/step - loss: 0.0546 - accuracy: 0.9847 - val_loss: 1.1605 - val_accuracy: 0.7543
Epoch 76/100
210/210 [==============================] - 2s 9ms/step - loss: 0.0528 - accuracy: 0.9851 - val_loss: 1.1416 - val_accuracy: 0.7579
Epoch 77/100
210/210 [==============================] - 2s 9ms/step - loss: 0.0630 - accuracy: 0.9802 - val_loss: 1.2191 - val_accuracy: 0.7522
Epoch 78/100
210/210 [==============================] - 2s 9ms/step - loss: 0.0535 - accuracy: 0.9839 - val_loss: 1.1346 - val_accuracy: 0.7589
Epoch 79/100
210/210 [==============================] - 2s 9ms/step - loss: 0.0510 - accuracy: 0.9848 - val_loss: 1.2660 - val_accuracy: 0.7481
Epoch 80/100
210/210 [==============================] - 2s 9ms/step - loss: 0.0583 - accuracy: 0.9827 - val_loss: 1.2543 - val_accuracy: 0.7339
Epoch 81/100
210/210 [==============================] - 2s 9ms/step - loss: 0.0609 - accuracy: 0.9823 - val_loss: 1.3351 - val_accuracy: 0.7294
Epoch 82/100
210/210 [==============================] - 2s 9ms/step - loss: 0.0525 - accuracy: 0.9840 - val_loss: 1.2451 - val_accuracy: 0.7429
Epoch 83/100
210/210 [==============================] - 2s 9ms/step - loss: 0.0644 - accuracy: 0.9806 - val_loss: 1.1909 - val_accuracy: 0.7568
Epoch 84/100
210/210 [==============================] - 2s 9ms/step - loss: 0.0505 - accuracy: 0.9846 - val_loss: 1.2577 - val_accuracy: 0.7387
Epoch 85/100
210/210 [==============================] - 2s 9ms/step - loss: 0.0469 - accuracy: 0.9862 - val_loss: 1.3195 - val_accuracy: 0.7282
Epoch 86/100
210/210 [==============================] - 2s 9ms/step - loss: 0.0483 - accuracy: 0.9861 - val_loss: 1.2324 - val_accuracy: 0.7523
Epoch 87/100
210/210 [==============================] - 2s 9ms/step - loss: 0.0537 - accuracy: 0.9837 - val_loss: 1.1874 - val_accuracy: 0.7551
Epoch 88/100
210/210 [==============================] - 2s 9ms/step - loss: 0.0528 - accuracy: 0.9837 - val_loss: 1.1167 - val_accuracy: 0.7698
Epoch 89/100
210/210 [==============================] - 2s 9ms/step - loss: 0.0487 - accuracy: 0.9853 - val_loss: 1.4038 - val_accuracy: 0.7184
Epoch 90/100
210/210 [==============================] - 2s 9ms/step - loss: 0.0477 - accuracy: 0.9860 - val_loss: 1.2795 - val_accuracy: 0.7429
Epoch 91/100
210/210 [==============================] - 2s 9ms/step - loss: 0.0482 - accuracy: 0.9860 - val_loss: 1.5518 - val_accuracy: 0.7071
Epoch 92/100
210/210 [==============================] - 2s 9ms/step - loss: 0.0442 - accuracy: 0.9871 - val_loss: 1.6883 - val_accuracy: 0.7047
Epoch 93/100
210/210 [==============================] - 2s 9ms/step - loss: 0.0466 - accuracy: 0.9859 - val_loss: 1.2722 - val_accuracy: 0.7533
Epoch 94/100
210/210 [==============================] - 2s 9ms/step - loss: 0.0490 - accuracy: 0.9851 - val_loss: 1.4654 - val_accuracy: 0.7151
Epoch 95/100
210/210 [==============================] - 2s 9ms/step - loss: 0.0419 - accuracy: 0.9875 - val_loss: 1.3203 - val_accuracy: 0.7382
Epoch 96/100
210/210 [==============================] - 2s 9ms/step - loss: 0.0436 - accuracy: 0.9866 - val_loss: 1.3964 - val_accuracy: 0.7347
Epoch 97/100
210/210 [==============================] - 2s 9ms/step - loss: 0.0569 - accuracy: 0.9820 - val_loss: 2.1019 - val_accuracy: 0.6540
Epoch 98/100
210/210 [==============================] - 2s 9ms/step - loss: 0.0516 - accuracy: 0.9850 - val_loss: 1.2758 - val_accuracy: 0.7477
Epoch 99/100
210/210 [==============================] - 2s 9ms/step - loss: 0.0425 - accuracy: 0.9872 - val_loss: 1.1916 - val_accuracy: 0.7586
Epoch 100/100
210/210 [==============================] - 2s 9ms/step - loss: 0.0383 - accuracy: 0.9888 - val_loss: 1.4676 - val_accuracy: 0.7302
In [123]:
results_final = keras_model_final.evaluate(X_test, y_test)
print('Test accuracy : ', results_final[1])
563/563 [==============================] - 3s 5ms/step - loss: 1.4676 - accuracy: 0.7302
Test accuracy :  0.730222225189209
In [130]:
print(history.history.keys())
# summarize history for accuracy
plt.plot(history_final.history['accuracy'])
plt.plot(history_final.history['val_accuracy'])
plt.title('model accuracy')
plt.ylabel('accuracy')
plt.xlabel('epoch')
plt.legend(['train', 'test'], loc='upper left')
plt.show()
# summarize history for loss
plt.plot(history_final.history['loss'])
plt.plot(history_final.history['val_loss'])
plt.title('model loss')
plt.ylabel('loss')
plt.xlabel('epoch')
plt.legend(['train', 'test'], loc='upper left')
plt.show()
dict_keys(['loss', 'accuracy', 'val_loss', 'val_accuracy'])

Predicting the digits using NN classifier on test data

Image 1

In [124]:
#Showing the image
plt.imshow(X_test[5].reshape(32,32),cmap='gray')
Out[124]:
<matplotlib.image.AxesImage at 0x7f757acfb610>
In [125]:
#Predicting the digits
#keras_model_final.predict_classes(X_test)[5]
#predict_x=model.predict(X_test) 
classes_x=np.argmax(keras_model_final.predict(X_test)[5])#,axis=1)
classes_x
Out[125]:
9

Image 2

In [126]:
#Showing the image
plt.imshow(X_test[10].reshape(32,32),cmap='gray')
Out[126]:
<matplotlib.image.AxesImage at 0x7f75f9940f10>

Image 3

In [127]:
#Predicting the digits
#keras_model_final.predict_classes(X_test)[112]
classes_x=np.argmax(keras_model_final.predict(X_test)[112])#,axis=1)
classes_x
Out[127]:
9
In [128]:
#Predicting for all images
y_pred= np.argmax(keras_model_final.predict(X_test),axis=1)
#keras_model_final.predict_classes(X_test)
print(y_pred)
[5 2 2 ... 7 9 2]

Differences and trade-offs between traditional KNN and NN classifiers with the help of classification metrics

  • KNN requires more time to find the nearest neibhours. It takes less time to train the model. It took more than 24hrs to find the best neighbour(k). However, Neural Network takes more time to train the model.
  • In KNN, we have only one hyperparameter k while in NN we have so many hyperparameters(Hidden Layers, Hidden Units,learning Rate, Number of Epochs, activations Functions).
  • KNN classification metrics shows that the accuracy is 46%. The digit 4 and 6 are classified correctly and achieved approx 60% accuracy. the other digits are not predicted correclty.
  • Using NN with batchNormalization we got 83.7% accuracy. We have predicted the classification of the images correctly.
  • used various methods in NN for ggetting better accuracy
  • facing memry outof bound error so used some samples so it is not upto mark models
In [129]:
results_on_val
Out[129]:
Method accuracy
1 NN 0.831300
2 NN_Batch_adam 0.818667
3 NN_Batch_sgd 0.129283
4 NN_Batch_Drop_adam 0.705233
In [ ]: